CIHealth
<
Single Day: Oct 28▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: DEV

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
2h 13m 0s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 8bf5e0d
 centralus n/a passed #7817 (open) 0 n/a
2h 10m 44s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 2a5af5f
 centralus e2e failed ⚠#7809 (open) 2
e2e (2)
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional t... message truncated; provider Microsoft.RedHatOpenShift
20:11:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0x4218f58d8c0>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-8q6szjwkq6rk" to finish creating: GET https://rp.j0183936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/78051d0a-b2ad-4647-99d4-120efa3f6a99
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/78051d0a-b2ad-4647-99d4-120efa3f6a99",
      "name": "78051d0a-b2ad-4647-99d4-120efa3f6a99",
      "status": "Failed",
      "startTime": "2026-10-28T21:13:11.691142567Z",
      "endTime": "2026-10-28T21:32:17.541374372Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a7p5v6a9s2f0e3j-nodepool-128gib-brx87-m46c2: Failed: interfaces failed to create or update. err: failed to get existing resource clusternp128-8q6szjwkq6rk--managed/a7p5v6a9s2f0e3j-nodepool-128gib-brx87-m46c2-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-8q6szjwkq6rk--managed/providers/Microsoft.Network/networkInterfaces/a7p5v6a9s2f0e3j-nodepool-128gib-brx87-m46c2-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3996e871-9d8c-4863-9b56-cf4685281cdc' exceeded. Please try again after '10' seconds after additional t... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0183936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/78051d0a-b2ad-4647-99d4-120efa3f6a99
            correlation ID: 7ef5aa4f-453a-4950-9b06-46a903ca993d
            client request ID: a9dcefcf-ba4d-4111-9c80-6dbba9abd781
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:96]: failed to create node pool "nodepool-128GiB" with 128GiB osDisk
Unexpected error:
    <*fmt.wrapError | 0x4218f58d8c0>: 
    failed to create NodePool nodepool-128GiB: failed waiting for nodepool="nodepool-128GiB" for cluster "hcp-cluster-np-128" in resourcegroup="clusternp128-8q6szjwkq6rk" to finish creating: GET https://rp.j0183936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/78051d0a-b2ad-4647-99d4-120efa3f6a99
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/78051d0a-b2ad-4647-99d4-120efa3f6a99",
      "name": "78051d0a-b2ad-4647-99d4-120efa3f6a99",
      "status": "Failed",
      "startTime": "2026-10-28T21:13:11.691142567Z",
      "endTime": "2026-10-28T21:32:17.541374372Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a7p5v6a9s2f0e3j-nodepool-128gib-brx87-m46c2: Failed: interfaces failed to create or update. err: failed to get existing resource clusternp128-8q6szjwkq6rk--managed/a7p5v6a9s2f0e3j-nodepool-128gib-brx87-m46c2-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-8q6szjwkq6rk--managed/providers/Microsoft.Network/networkInterfaces/a7p5v6a9s2f0e3j-nodepool-128gib-brx87-m46c2-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '3996e871-9d8c-4863-9b56-cf4685281cdc' exceeded. Please try again after '10' seconds after additional t... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0183936.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/78051d0a-b2ad-4647-99d4-120efa3f6a99
            correlation ID: 7ef5aa4f-453a-4950-9b06-46a903ca993d
            client request ID: a9dcefcf-ba4d-4111-9c80-6dbba9abd781
occurred
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j0183936-mgmt-1/ocm-arohcpci01-<id>: expected
20:11:40 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hcf7x894jgzg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2424493382e0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hcf7x894jgzg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j0183936-mgmt-1/ocm-arohcpci01-2to1b0ie1a4878pqeni53tn93ag3ur6i-t8h4i5b2e7a9g5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hcf7x894jgzg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j0183936-mgmt-1/ocm-arohcpci01-2to1b0ie1a4878pqeni53tn93ag3ur6i-t8h4i5b2e7a9g5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x2424493345b0>{
            s: "dev/centralus/ci01-j0183936-mgmt-1/ocm-arohcpci01-2to1b0ie1a4878pqeni53tn93ag3ur6i-t8h4i5b2e7a9g5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hcf7x894jgzg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2424493382e0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hcf7x894jgzg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j0183936-mgmt-1/ocm-arohcpci01-2to1b0ie1a4878pqeni53tn93ag3ur6i-t8h4i5b2e7a9g5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-hcf7x894jgzg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: dev/centralus/ci01-j0183936-mgmt-1/ocm-arohcpci01-2to1b0ie1a4878pqeni53tn93ag3ur6i-t8h4i5b2e7a9g5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*errors.errorString | 0x2424493345b0>{
            s: "dev/centralus/ci01-j0183936-mgmt-1/ocm-arohcpci01-2to1b0ie1a4878pqeni53tn93ag3ur6i-t8h4i5b2e7a9g5f: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        },
    }
occurred
2h 16m 24s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 75e7373
 westus3 alert, e2e failed ⚠#7757 (open) 2
e2e (1)
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-bkpxxt' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network
19:56:50 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-7tbfp9"
Unexpected error:
    <*errors.joinError | 0x28672532e420>: 
    VerifyIngressDNSRecord(aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud) timed out after 10m0.005s: DNS A record "*.e2e-dns-bkpxxt" not yet present in zone rg-candidate-4-23-cwv9mngrfv9r-candidate-7tbfp9--managed/aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-cwv9mngrfv9r-candidate-7tbfp9--managed/providers/Microsoft.Network/dnsZones/aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud/A/*.e2e-dns-bkpxxt
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-bkpxxt' does not exist in resource group 'rg-candidate-4-23-cwv9mngrfv9r-candidate-7tbfp9--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-cwv9mngrfv9r-candidate-7tbfp9--managed/providers/Microsoft.Network/dnsZones/aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud/A/*.e2e-dns-bkpxxt
            correlation ID: 00f470eb-29ca-4c55-b41f-faa8336d52a1
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-7tbfp9"
Unexpected error:
    <*errors.joinError | 0x28672532e420>: 
    VerifyIngressDNSRecord(aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud) timed out after 10m0.005s: DNS A record "*.e2e-dns-bkpxxt" not yet present in zone rg-candidate-4-23-cwv9mngrfv9r-candidate-7tbfp9--managed/aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-cwv9mngrfv9r-candidate-7tbfp9--managed/providers/Microsoft.Network/dnsZones/aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud/A/*.e2e-dns-bkpxxt
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-bkpxxt' does not exist in resource group 'rg-candidate-4-23-cwv9mngrfv9r-candidate-7tbfp9--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-cwv9mngrfv9r-candidate-7tbfp9--managed/providers/Microsoft.Network/dnsZones/aro.r8a3f2x2r3m7g7z.8i1p.j8906368.hcp.osadev.cloud/A/*.e2e-dns-bkpxxt
            correlation ID: 00f470eb-29ca-4c55-b41f-faa8336d52a1
            client request ID: unavailable
occurred
alert (1)
alert [svc] BackendControllerQueueDepthHigh fired
19:56:50 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T21:09:13Z
  Ended:       2026-10-28T21:26:12Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8906368-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
22m 45s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 0bb0b65
 n/a unknown failed #7693 (open) 1 non-artifact-backed failure (no junit artifacts)
2h 8m 17s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head ffe9a1a
 centralus n/a passed #7787 (open) 0 n/a
1h 16m 2s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 0fbb7c0
 canadacentral unknown failed #7651 (open) 1 non-artifact-backed failure (no junit artifacts)
2h 32m 26s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head f46d8ee
 westus3 alert, e2e failed #7666 (open) 4
e2e (1)
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
19:15:51 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x3a2a28c8a260>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-h8dst6v865z6" to finish creating: GET https://rp.j1157504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff",
      "name": "455ab4ea-ab03-4a58-91c6-5ad5406405ff",
      "status": "Failed",
      "startTime": "2026-10-28T20:16:02.73250898Z",
      "endTime": "2026-10-28T20:45:04.251585998Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1157504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff
            correlation ID: 582a61df-2172-42e6-962b-b544533d3da6
            client request ID: a46122fb-f2eb-46f8-9be1-73dd5a442cce
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x3a2a28c8a260>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-h8dst6v865z6" to finish creating: GET https://rp.j1157504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff",
      "name": "455ab4ea-ab03-4a58-91c6-5ad5406405ff",
      "status": "Failed",
      "startTime": "2026-10-28T20:16:02.73250898Z",
      "endTime": "2026-10-28T20:45:04.251585998Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1157504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/455ab4ea-ab03-4a58-91c6-5ad5406405ff
            correlation ID: 582a61df-2172-42e6-962b-b544533d3da6
            client request ID: a46122fb-f2eb-46f8-9be1-73dd5a442cce
occurred
alert (3)
alert [hcp] KubeStatefulSetReplicasMismatch fired
19:15:51 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T20:37:24Z
  Ended:       2026-10-28T21:04:22Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-28T20:37:24Z
  Ended:       2026-10-28T21:01:24Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert [hcp] KubePodCrashLooping fired
19:15:51 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T20:39:20Z
  Ended:       2026-10-28T21:05:19Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="777a1913-78dc-4e11-9844-4f50d6675327"
  Description: Pod ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-28T20:40:18Z
  Ended:       2026-10-28T21:06:20Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="777a1913-78dc-4e11-9844-4f50d6675327"
  Description: Pod ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
alert [hcp] KubeDeploymentReplicasMismatch fired
19:15:51 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T20:52:20Z
  Ended:       2026-10-28T21:03:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-28T20:52:20Z
  Ended:       2026-10-28T21:03:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1157504-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.172:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1157504/providers/microsoft.monitor/accounts/hcps-j1157504", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-p48cd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2to0k0skbecjdmuid7733ieqr7h7trvg-private-kas/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
2h 28m 33s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head c17975b
 westus3 n/a passed #7802 (open) 0 n/a
27m 19s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 2a5af5f
 n/a unknown failed #7809 (open) 1 non-artifact-backed failure (no junit artifacts)
2h 11m 54s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 84f443c
 centralus alert, e2e failed #7764 (open) 6
e2e (3)
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
18:33:36 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0x370d5101eb20>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-tgjxrh8mcrw6" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb",
      "name": "cce1648d-4b31-48bb-9884-f170bb5391eb",
      "status": "Failed",
      "startTime": "2026-10-28T19:28:55.409225524Z",
      "endTime": "2026-10-28T19:58:03.222551854Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb
            correlation ID: d6e5235d-8c3c-4404-82e9-f3e3b3165e1e
            client request ID: 4d040942-ab41-4fbc-8d6e-d69a84105ef1
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0x370d5101eb20>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-tgjxrh8mcrw6" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb",
      "name": "cce1648d-4b31-48bb-9884-f170bb5391eb",
      "status": "Failed",
      "startTime": "2026-10-28T19:28:55.409225524Z",
      "endTime": "2026-10-28T19:58:03.222551854Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce1648d-4b31-48bb-9884-f170bb5391eb
            correlation ID: d6e5235d-8c3c-4404-82e9-f3e3b3165e1e
            client request ID: 4d040942-ab41-4fbc-8d6e-d69a84105ef1
occurred
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
18:33:36 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-5-1-678df7"
Unexpected error:
    <*fmt.wrapError | 0x2d18a6e97e00>: 
    DNS for route host agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x2d18a6c74540>{
            msg: "DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:241]: failed to verify simple web app runs on cluster "cluster-candidate-5-1-678df7"
Unexpected error:
    <*fmt.wrapError | 0x2d18a6e97e00>: 
    DNS for route host agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x2d18a6c74540>{
            msg: "DNS for agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-vqnk2.apps.aro.c7e9m0p1o3l4d4j.d5kj.j8717568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift
18:33:36 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0x740ee223cf8>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-dmmfjzfldpm8" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee",
      "name": "e51bb1e4-4996-454f-a81d-9354613a75ee",
      "status": "Failed",
      "startTime": "2026-10-28T19:36:46.639467207Z",
      "endTime": "2026-10-28T19:55:53.152282953Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-8z842-x96hm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cl-cilium-np-8z842-tlsx7: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-dmmfjzfldpm8--managed/cilium-cl-cilium-np-8z842-tlsx7-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-dmmfjzfldpm8--managed/providers/Microsoft.Network/networkInterfaces/cilium-cl-cilium-np-8z842-tlsx7-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '1d2f4929-515e-4e8d-8580-836e819f8d50' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
            correlation ID: 4963ae88-5109-4671-873f-f28a9deccdac
            client request ID: 38a19ead-d2a2-4130-9e04-1364166950d2
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0x740ee223cf8>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-dmmfjzfldpm8" to finish creating: GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee",
      "name": "e51bb1e4-4996-454f-a81d-9354613a75ee",
      "status": "Failed",
      "startTime": "2026-10-28T19:36:46.639467207Z",
      "endTime": "2026-10-28T19:55:53.152282953Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine cilium-cl-cilium-np-8z842-x96hm: InspectionFailed: Waiting for AzureMachine to report spec.providerID\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cl-cilium-np-8z842-tlsx7: Failed: interfaces failed to create or update. err: failed to get existing resource cni-cilium-dmmfjzfldpm8--managed/cilium-cl-cilium-np-8z842-tlsx7-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-dmmfjzfldpm8--managed/providers/Microsoft.Network/networkInterfaces/cilium-cl-cilium-np-8z842-tlsx7-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '1d2f4929-515e-4e8d-8580-836e819f8d50' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https://aka... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8717568.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e51bb1e4-4996-454f-a81d-9354613a75ee
            correlation ID: 4963ae88-5109-4671-873f-f28a9deccdac
            client request ID: 38a19ead-d2a2-4130-9e04-1364166950d2
occurred
alert (3)
alert [hcp] KubeDeploymentReplicasMismatch fired
18:33:36 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T20:03:50Z
  Ended:       2026-10-28T20:15:46Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-59dhx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-28T20:03:50Z
  Ended:       2026-10-28T20:15:46Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-59dhx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert [hcp] KubeStatefulSetReplicasMismatch fired
18:33:36 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T19:48:54Z
  Ended:       2026-10-28T20:14:47Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-59dhx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-28T19:48:54Z
  Ended:       2026-10-28T20:14:47Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-59dhx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert [hcp] KubePodCrashLooping fired
18:33:36 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T19:50:44Z
  Ended:       2026-10-28T20:18:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="aad698ce-6b47-41d1-8ddd-e6e4cb15b5d7"
  Description: Pod ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-28T19:50:44Z
  Ended:       2026-10-28T20:19:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8717568-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.254:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8717568/providers/microsoft.monitor/accounts/hcps-j8717568", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="aad698ce-6b47-41d1-8ddd-e6e4cb15b5d7"
  Description: Pod ocm-arohcpci01-2tnvtu6q95ufl5426ec9nsjbnsqfa0aa-a8y2r7f6q8h7z0t/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
2h 21m 39s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 46a7b95
 canadacentral e2e failed ⚠#7816 (open) 2
e2e (2)
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try ag... message truncated; provider Microsoft.RedHatOpenShift
18:33:03 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x3f06fbed83a0>: 
    failed to create NodePool swift-workers: failed waiting for nodepool="swift-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-4lmptk5vpx7s" to finish creating: GET https://rp.j7626240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4b98636-6475-41ec-a151-007f2ef479ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4b98636-6475-41ec-a151-007f2ef479ea",
      "name": "f4b98636-6475-41ec-a151-007f2ef479ea",
      "status": "Failed",
      "startTime": "2026-10-28T19:44:01.234316083Z",
      "endTime": "2026-10-28T20:03:07.974415443Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l3j9w2r9y4j1z4u-swift-workers-4vqnr-xznbb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4lmptk5vpx7s--managed/providers/Microsoft.Compute/virtualMachines/l3j9w2r9y4j1z4u-swift-workers-4vqnr-xznbb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l3j9w2r9y4j1z4u-swift-workers-4vqnr-xznbb: Failed: availabilitysets failed to create or update. err: failed to get existing resource kusto-logs-4lmptk5vpx7s--managed/2tnvvns3jn8rgfrctgfo79mkpouiqrqc_l3j9w2r9y4j1z4u-swift-workers-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4lmptk5vpx7s--managed/providers/Microsoft.Compute/availabilitySets/2tnvvns3jn8rgfrctgfo79mkpouiqrqc_l3j9w2r9y4j1z4u-swift-workers-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '1868e77e-aae0-4284-940e-e4cd46ae5baa' exceeded. Please try ag... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7626240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4b98636-6475-41ec-a151-007f2ef479ea
            correlation ID: 1f8c9a64-07bf-44f3-bca5-a4308cdb0f56
            client request ID: 49aeda1c-19cc-4647-af7d-1d78f910596d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x3f06fbed83a0>: 
    failed to create NodePool swift-workers: failed waiting for nodepool="swift-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-4lmptk5vpx7s" to finish creating: GET https://rp.j7626240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4b98636-6475-41ec-a151-007f2ef479ea
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4b98636-6475-41ec-a151-007f2ef479ea",
      "name": "f4b98636-6475-41ec-a151-007f2ef479ea",
      "status": "Failed",
      "startTime": "2026-10-28T19:44:01.234316083Z",
      "endTime": "2026-10-28T20:03:07.974415443Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine l3j9w2r9y4j1z4u-swift-workers-4vqnr-xznbb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4lmptk5vpx7s--managed/providers/Microsoft.Compute/virtualMachines/l3j9w2r9y4j1z4u-swift-workers-4vqnr-xznbb to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine l3j9w2r9y4j1z4u-swift-workers-4vqnr-xznbb: Failed: availabilitysets failed to create or update. err: failed to get existing resource kusto-logs-4lmptk5vpx7s--managed/2tnvvns3jn8rgfrctgfo79mkpouiqrqc_l3j9w2r9y4j1z4u-swift-workers-as (service: availabilitysets): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-4lmptk5vpx7s--managed/providers/Microsoft.Compute/availabilitySets/2tnvvns3jn8rgfrctgfo79mkpouiqrqc_l3j9w2r9y4j1z4u-swift-workers-as\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '1868e77e-aae0-4284-940e-e4cd46ae5baa' exceeded. Please try ag... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7626240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4b98636-6475-41ec-a151-007f2ef479ea
            correlation ID: 1f8c9a64-07bf-44f3-bca5-a4308cdb0f56
            client request ID: 49aeda1c-19cc-4647-af7d-1d78f910596d
occurred
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not
18:33:03 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-nxx9xr"
Unexpected error:
    <*errors.joinError | 0x138a2d1f8048>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb": nodes "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x138a2cdf5020>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x138a2cdf5000>{
                    msg: "failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x138a2cdf4fe0>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x138a2cd0dcc0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-nxx9xr"
Unexpected error:
    <*errors.joinError | 0x138a2d1f8048>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb": nodes "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x138a2cdf5020>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x138a2cdf5000>{
                    msg: "failed to label node e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x138a2cdf4fe0>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\": nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x138a2cd0dcc0>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "e3a5t5r1l8c7f8q-np-nxx9xr-qn466-6zltb",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
2h 20m 13s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head c44baa7 · merge c44baa7
 westus3 n/a passed #7537 (merged) 0 n/a
2h 27m 39s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 6fc7a87
 westus3 alert, e2e failed ⚠#7785 (open) 2
e2e (1)
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-czbqgt' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network
18:23:09 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-kzwwft"
Unexpected error:
    <*errors.joinError | 0x2fea822a5d0>: 
    VerifyIngressDNSRecord(aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud) timed out after 10m0s: DNS A record "*.e2e-dns-czbqgt" not yet present in zone rg-candidate-4-23-qkhkjtrdhq8n-candidate-kzwwft--managed/aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-qkhkjtrdhq8n-candidate-kzwwft--managed/providers/Microsoft.Network/dnsZones/aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud/A/*.e2e-dns-czbqgt
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-czbqgt' does not exist in resource group 'rg-candidate-4-23-qkhkjtrdhq8n-candidate-kzwwft--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-qkhkjtrdhq8n-candidate-kzwwft--managed/providers/Microsoft.Network/dnsZones/aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud/A/*.e2e-dns-czbqgt
            correlation ID: ef507981-8069-4371-b52a-e18fb184cf37
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-kzwwft"
Unexpected error:
    <*errors.joinError | 0x2fea822a5d0>: 
    VerifyIngressDNSRecord(aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud) timed out after 10m0s: DNS A record "*.e2e-dns-czbqgt" not yet present in zone rg-candidate-4-23-qkhkjtrdhq8n-candidate-kzwwft--managed/aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-qkhkjtrdhq8n-candidate-kzwwft--managed/providers/Microsoft.Network/dnsZones/aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud/A/*.e2e-dns-czbqgt
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-czbqgt' does not exist in resource group 'rg-candidate-4-23-qkhkjtrdhq8n-candidate-kzwwft--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-qkhkjtrdhq8n-candidate-kzwwft--managed/providers/Microsoft.Network/dnsZones/aro.m3u7u2n3l0l3p3b.0iw8.j3991936.hcp.osadev.cloud/A/*.e2e-dns-czbqgt
            correlation ID: ef507981-8069-4371-b52a-e18fb184cf37
            client request ID: unavailable
occurred
alert (1)
alert [svc] BackendControllerQueueDepthHigh fired
18:23:09 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T19:42:52Z
  Ended:       2026-10-28T20:02:53Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3991936-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
2h 22m 0s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 2f691cf
 westus3 alert, e2e failed ★#7814 (open) 2
e2e (1)
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: no customer resource ID -> live ReadDesire HostedCluster ->
18:22:59 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w96mrp6lpnz7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x67403d76ae0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w96mrp6lpnz7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w96mrp6lpnz7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0x67403d76ac0>{
            observation: <*errors.errorString | 0x674032428b0>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0x67403d76aa0>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j5001984-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnvsselnr4pgg1e0n2skr2pggevm4b6-w8s3q8g2g8v3s0d\"; let start = datetime(2026-10-28T19:24:48.1549118Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x674036d06e0>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x67403d76a80>{
                        msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j5001984-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnvsselnr4pgg1e0n2skr2pggevm4b6-w8s3q8g2g8v3s0d\"; let start = datetime(2026-10-28T19:24:48.1549118Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x67404ab4ba0>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w96mrp6lpnz7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x67403d76ae0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w96mrp6lpnz7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-w96mrp6lpnz7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.004s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0x67403d76ac0>{
            observation: <*errors.errorString | 0x674032428b0>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0x67403d76aa0>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j5001984-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnvsselnr4pgg1e0n2skr2pggevm4b6-w8s3q8g2g8v3s0d\"; let start = datetime(2026-10-28T19:24:48.1549118Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x674036d06e0>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x67403d76a80>{
                        msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j5001984-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnvsselnr4pgg1e0n2skr2pggevm4b6-w8s3q8g2g8v3s0d\"; let start = datetime(2026-10-28T19:24:48.1549118Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x67404ab4ba0>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
alert (1)
alert [hcp] KubePodNotReady fired
18:22:59 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T20:00:50Z
  Ended:       2026-10-28T20:06:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5001984-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tnvsms9l35e1dde1cr2bnekd5epm5j9-ingress-cert-le", pod="openshift-apiserver-7cb4d98fc-rzx96", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tnvsms9l35e1dde1cr2bnekd5epm5j9-ingress-cert-le/openshift-apiserver-7cb4d98fc-rzx96 has been in a non-ready state for longer than 5 minutes.
2h 33m 33s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head e7c23ba
 westus3 e2e failed ⚠#7704 (open) 2
e2e (2)
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: no customer resource ID -> live ReadDesire HostedCluster ->
18:12:04 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7qgq6lcld5tj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x31615201e9a0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7qgq6lcld5tj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7qgq6lcld5tj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0x31615201e980>{
            observation: <*errors.errorString | 0x316153080da0>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0x31615201e960>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9822336-mgmt-1\"; let ns = \"ocm-arohcpci01-2tnvqtddfc0hnmn52ol3g2lfbm6i22ri-i7y4b4h2z7o4a5h\"; let start = datetime(2026-10-28T19:21:43.9974261Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x316150f60aa0>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x31615201e920>{
                        msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9822336-mgmt-1\"; let ns = \"ocm-arohcpci01-2tnvqtddfc0hnmn52ol3g2lfbm6i22ri-i7y4b4h2z7o4a5h\"; let start = datetime(2026-10-28T19:21:43.9974261Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x316152c10a80>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7qgq6lcld5tj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x31615201e9a0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7qgq6lcld5tj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-7qgq6lcld5tj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.005s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0x31615201e980>{
            observation: <*errors.errorString | 0x316153080da0>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0x31615201e960>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9822336-mgmt-1\"; let ns = \"ocm-arohcpci01-2tnvqtddfc0hnmn52ol3g2lfbm6i22ri-i7y4b4h2z7o4a5h\"; let start = datetime(2026-10-28T19:21:43.9974261Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x316150f60aa0>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x31615201e920>{
                        msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9822336-mgmt-1\"; let ns = \"ocm-arohcpci01-2tnvqtddfc0hnmn52ol3g2lfbm6i22ri-i7y4b4h2z7o4a5h\"; let start = datetime(2026-10-28T19:21:43.9974261Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x316152c10a80>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
18:12:04 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-jl2sj8 with version 4.21.40
Unexpected error:
    <*fmt.wrapError | 0x2ee0c1834d20>: 
    failed to create HCP cluster np-version-upgrade-cluster-jl2sj8: failed waiting for cluster="np-version-upgrade-cluster-jl2sj8" in resourcegroup="rg-np-version-upgrade-jl2sj8-22vp9qvqkmcf" to finish creating: GET https://rp.j9822336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583",
      "name": "267d3daf-f468-46eb-af3a-f1b72d573583",
      "status": "Failed",
      "startTime": "2026-10-28T19:42:27.649991944Z",
      "endTime": "2026-10-28T20:11:28.996573337Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9822336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583
            correlation ID: c45a1c96-f9ee-44e0-8639-c952cdb59a23
            client request ID: fe15252c-edf3-4b0e-8aaa-b5cd06393883
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-jl2sj8 with version 4.21.40
Unexpected error:
    <*fmt.wrapError | 0x2ee0c1834d20>: 
    failed to create HCP cluster np-version-upgrade-cluster-jl2sj8: failed waiting for cluster="np-version-upgrade-cluster-jl2sj8" in resourcegroup="rg-np-version-upgrade-jl2sj8-22vp9qvqkmcf" to finish creating: GET https://rp.j9822336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583",
      "name": "267d3daf-f468-46eb-af3a-f1b72d573583",
      "status": "Failed",
      "startTime": "2026-10-28T19:42:27.649991944Z",
      "endTime": "2026-10-28T20:11:28.996573337Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9822336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/267d3daf-f468-46eb-af3a-f1b72d573583
            correlation ID: c45a1c96-f9ee-44e0-8639-c952cdb59a23
            client request ID: fe15252c-edf3-4b0e-8aaa-b5cd06393883
occurred
2h 31m 56s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 6405f41
 centralus e2e failed ⚠#6688 (open) 1
e2e (1)
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-p2v629' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network
17:45:00 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-r5cltn"
Unexpected error:
    <*errors.joinError | 0x263db2a8cd50>: 
    VerifyIngressDNSRecord(aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud) timed out after 10m0.003s: DNS A record "*.e2e-dns-p2v629" not yet present in zone rg-candidate-4-23-6vt99sh5dqhr-candidate-r5cltn--managed/aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-6vt99sh5dqhr-candidate-r5cltn--managed/providers/Microsoft.Network/dnsZones/aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud/A/*.e2e-dns-p2v629
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-p2v629' does not exist in resource group 'rg-candidate-4-23-6vt99sh5dqhr-candidate-r5cltn--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-6vt99sh5dqhr-candidate-r5cltn--managed/providers/Microsoft.Network/dnsZones/aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud/A/*.e2e-dns-p2v629
            correlation ID: 0eb3f5e1-8fdc-4f54-ad01-280d07fdef7f
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-r5cltn"
Unexpected error:
    <*errors.joinError | 0x263db2a8cd50>: 
    VerifyIngressDNSRecord(aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud) timed out after 10m0.003s: DNS A record "*.e2e-dns-p2v629" not yet present in zone rg-candidate-4-23-6vt99sh5dqhr-candidate-r5cltn--managed/aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-6vt99sh5dqhr-candidate-r5cltn--managed/providers/Microsoft.Network/dnsZones/aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud/A/*.e2e-dns-p2v629
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-p2v629' does not exist in resource group 'rg-candidate-4-23-6vt99sh5dqhr-candidate-r5cltn--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-6vt99sh5dqhr-candidate-r5cltn--managed/providers/Microsoft.Network/dnsZones/aro.u8t8x8m6x2i2q9i.jz74.j3603072.hcp.osadev.cloud/A/*.e2e-dns-p2v629
            correlation ID: 0eb3f5e1-8fdc-4f54-ad01-280d07fdef7f
            client request ID: unavailable
occurred
2h 4m 9s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head e2aa104 · merge e2aa104
 westus3 n/a passed #7705 (merged) 0 n/a
2h 19m 46s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 1c8be18
 canadacentral e2e failed #7811 (open) 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.... message truncated; provider Microsoft.RedHatOpenShift
16:59:20 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc2b6fc92f40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-4pl8j4mzdr2l" to finish creating: GET https://rp.j0412544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67bb2317-9c7e-4d3b-8e3c-0ad00e76a125
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67bb2317-9c7e-4d3b-8e3c-0ad00e76a125",
      "name": "67bb2317-9c7e-4d3b-8e3c-0ad00e76a125",
      "status": "Failed",
      "startTime": "2026-10-28T18:04:23.346337134Z",
      "endTime": "2026-10-28T18:23:31.470906003Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v2e1t9h4b7t5u0a-np-1-cd7br-b9qvn: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v2e1t9h4b7t5u0a-np-1-cd7br-b9qvn: Failed: interfaces failed to create or update. err: failed to get existing resource autoscaling-cluster-4pl8j4mzdr2l--managed/v2e1t9h4b7t5u0a-np-1-cd7br-b9qvn-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-4pl8j4mzdr2l--managed/providers/Microsoft.Network/networkInterfaces/v2e1t9h4b7t5u0a-np-1-cd7br-b9qvn-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'f0916b6b-4fcf-4b8b-b7f8-5a66898b1190' exceeded. Please try again after '2' seconds after additional tokens are available.... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0412544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67bb2317-9c7e-4d3b-8e3c-0ad00e76a125
            correlation ID: 7d414729-3b90-4c4b-9628-b2993ebd631c
            client request ID: 0c6fdc27-946b-4c40-9ac6-c7629afa8ed5
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc2b6fc92f40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-4pl8j4mzdr2l" to finish creating: GET https://rp.j0412544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67bb2317-9c7e-4d3b-8e3c-0ad00e76a125
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67bb2317-9c7e-4d3b-8e3c-0ad00e76a125",
      "name": "67bb2317-9c7e-4d3b-8e3c-0ad00e76a125",
      "status": "Failed",
      "startTime": "2026-10-28T18:04:23.346337134Z",
      "endTime": "2026-10-28T18:23:31.470906003Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v2e1t9h4b7t5u0a-np-1-cd7br-b9qvn: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v2e1t9h4b7t5u0a-np-1-cd7br-b9qvn: Failed: interfaces failed to create or update. err: failed to get existing resource autoscaling-cluster-4pl8j4mzdr2l--managed/v2e1t9h4b7t5u0a-np-1-cd7br-b9qvn-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-4pl8j4mzdr2l--managed/providers/Microsoft.Network/networkInterfaces/v2e1t9h4b7t5u0a-np-1-cd7br-b9qvn-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'f0916b6b-4fcf-4b8b-b7f8-5a66898b1190' exceeded. Please try again after '2' seconds after additional tokens are available.... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0412544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/67bb2317-9c7e-4d3b-8e3c-0ad00e76a125
            correlation ID: 7d414729-3b90-4c4b-9628-b2993ebd631c
            client request ID: 0c6fdc27-946b-4c40-9ac6-c7629afa8ed5
occurred
2h 20m 22s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 027de3a
 westus3 e2e failed ⚠#7808 (open) 1
e2e (1)
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j5676416-mgmt-2/ocm-arohcpci01-<id>
16:49:13 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dx26gclbnqgv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x3756ccac8820>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dx26gclbnqgv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j5676416-mgmt-2/ocm-arohcpci01-2tnufqipa4peho2pme6jiu9n7kj695rj-r8q8e3h9t3h8q1o router=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000: expected matching summary after test start and within 8m; summaryUID=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000 pass=5eb81438-b4f4-4d92-b379-a83081a8b2cc at=2026-10-28T18:18:30Z
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dx26gclbnqgv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j5676416-mgmt-2/ocm-arohcpci01-2tnufqipa4peho2pme6jiu9n7kj695rj-r8q8e3h9t3h8q1o router=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000: expected matching summary after test start and within 8m; summaryUID=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000 pass=5eb81438-b4f4-4d92-b379-a83081a8b2cc at=2026-10-28T18:18:30Z",
        err: <*errors.errorString | 0x3756cb62d010>{
            s: "dev/westus3/ci01-j5676416-mgmt-2/ocm-arohcpci01-2tnufqipa4peho2pme6jiu9n7kj695rj-r8q8e3h9t3h8q1o router=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000: expected matching summary after test start and within 8m; summaryUID=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000 pass=5eb81438-b4f4-4d92-b379-a83081a8b2cc at=2026-10-28T18:18:30Z",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dx26gclbnqgv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x3756ccac8820>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dx26gclbnqgv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j5676416-mgmt-2/ocm-arohcpci01-2tnufqipa4peho2pme6jiu9n7kj695rj-r8q8e3h9t3h8q1o router=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000: expected matching summary after test start and within 8m; summaryUID=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000 pass=5eb81438-b4f4-4d92-b379-a83081a8b2cc at=2026-10-28T18:18:30Z
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dx26gclbnqgv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j5676416-mgmt-2/ocm-arohcpci01-2tnufqipa4peho2pme6jiu9n7kj695rj-r8q8e3h9t3h8q1o router=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000: expected matching summary after test start and within 8m; summaryUID=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000 pass=5eb81438-b4f4-4d92-b379-a83081a8b2cc at=2026-10-28T18:18:30Z",
        err: <*errors.errorString | 0x3756cb62d010>{
            s: "dev/westus3/ci01-j5676416-mgmt-2/ocm-arohcpci01-2tnufqipa4peho2pme6jiu9n7kj695rj-r8q8e3h9t3h8q1o router=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000: expected matching summary after test start and within 8m; summaryUID=8004b169-b285-4616-88a5-3129e6cda029 node=aks-userswft2-37523794-vmss000000 pass=5eb81438-b4f4-4d92-b379-a83081a8b2cc at=2026-10-28T18:18:30Z",
        },
    }
occurred
2h 13m 58s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head b79a3e3
 westus3 n/a passed #6951 (open) 0 n/a
1h 1m 41s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 8131773
 canadacentral alert, provision failed ⚠#7311 (open) 2
provision (1)
failed to roll out Helm release: resource Deployment/clusters-service/clusters-service not ready. status: InProgress, message: Available: 2/3 context deadline exceeded
16:20:01 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.ClusterService/service/deploy · suite templatize-pipeline
Show raw failure
time=2026-10-28T17:01:13.002Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-10-28T17:01:13.550Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy values="map[azureArmHelperIdentityCertName:armHelperPoolCert-25 azureArmHelperIdentityClientId:ea5e9c6c-1933-464d-8bf6-db343c22b92d azureArmHelperMockFpaPrincipalId:47f69502-0065-4d9a-b19b-d403e183d2f4 azureCloudEnvironmentName:AzurePublicCloud azureCsMiClientId:ae6cf536-acc0-414c-ada8-f268aeeb97e3 azureFirstPartyApplicationClientId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX azureMiMockServicePrincipalCertName:msiMockPoolCert-17 azureMiMockServicePrincipalClientId:e7112f7f-5b45-480e-bc44-abf9f219e6f0 azureMiMockServicePrincipalPrincipalId:39eefc4f-5e24-491d-9f09-bcc0ad573bcf azureOperatorsMI:map[dev:map[cloudControllerManager:map[roleDefinitions:[map[id:a1f96423-95ce-4224-ab27-4e3dc72facd4 name:Azure Red Hat OpenShift Cloud Controller Manager]]] cloudNetworkConfig:map[roleDefinitions:[map[id:be7a6435-15ae-4171-8f30-4a343eff9e8f name:Azure Red Hat OpenShift Network Operator]]] clusterApiAzure:map[roleDefinitions:[map[id:88366f10-ed47-4cc0-9fab-c8a06148393e name:Azure Red Hat OpenShift Hosted Control Planes Cluster API Provider]]] controlPlane:map[roleDefinitions:[map[id:fc0c873f-45e9-4d0d-a7d1-585aab30c6ed name:Azure Red Hat OpenShift Hosted Control Planes Control Plane Operator]]] diskCsiDriver:map[roleDefinitions:[map[id:5b7237c5-45e1-49d6-bc18-a1f62f400748 name:Azure Red Hat OpenShift Disk Storage Operator]]] fileCsiDriver:map[roleDefinitions:[map[id:0d7aedc0-15fd-4a67-a412-efad370c947e name:Azure Red Hat OpenShift File Storage Operator]]] imageRegistry:map[roleDefinitions:[map[id:8b32b316-c2f5-4ddf-b05b-83dacd2d08b5 name:Azure Red Hat OpenShift Image Registry Operator]]] ingress:map[roleDefinitions:[map[id:0336e1d3-7a87-462b-b6db-342b63f7802c name:Azure Red Hat OpenShift Cluster Ingress Operator]]] kms:map[roleDefinitions:[map[id:12338af0-0e69-4776-bea7-57ae8d297424 name:Key Vault Crypto User]]] serviceManagedIdentity:map[roleDefinitions:[map[id:c0ff367d-66d8-445e-917c-583feb0ef0d4 name:Azure Red Hat OpenShift Hosted Control Planes Service Managed Identity]]]] public:map[cloudControllerManager:map[roleDefinitions:[map[id:a1f96423-95ce-4224-ab27-4e3dc72facd4 name:Azure Red Hat OpenShift Cloud Controller Manager]]] cloudNetworkConfig:map[roleDefinitions:[map[id:be7a6435-15ae-4171-8f30-4a343eff9e8f name:Azure Red Hat OpenShift Network Operator]]] clusterApiAzure:map[roleDefinitions:[map[id:88366f10-ed47-4cc0-9fab-c8a06148393e name:Azure Red Hat OpenShift Hosted Control Planes Cluster API Provider]]] controlPlane:map[roleDefinitions:[map[id:fc0c873f-45e9-4d0d-a7d1-585aab30c6ed name:Azure Red Hat OpenShift Hosted Control Planes Control Plane Operator]]] diskCsiDriver:map[roleDefinitions:[map[id:5b7237c5-45e1-49d6-bc18-a1f62f400748 name:Azure Red Hat OpenShift Disk Storage Operator]]] fileCsiDriver:map[roleDefinitions:[map[id:0d7aedc0-15fd-4a67-a412-efad370c947e name:Azure Red Hat OpenShift File Storage Operator]]] imageRegistry:map[roleDefinitions:[map[id:8b32b316-c2f5-4ddf-b05b-83dacd2d08b5 name:Azure Red Hat OpenShift Image Registry Operator]]] ingress:map[roleDefinitions:[map[id:0336e1d3-7a87-462b-b6db-342b63f7802c name:Azure Red Hat OpenShift Cluster Ingress Operator]]] kms:map[roleDefinitions:[map[id:12338af0-0e69-4776-bea7-57ae8d297424 name:Key Vault Crypto User]]] serviceManagedIdentity:map[roleDefinitions:[map[id:c0ff367d-66d8-445e-917c-583feb0ef0d4 name:Azure Red Hat OpenShift Hosted Control Planes Service Managed Identity]]]] roleSetName:dev] backplaneURL: batchProcesses: batchProcessesDryRun:true clientId:foo clientScopes:openid clientSecret:bar clusterErrorReport:false cosmosDatabaseName:arohcpci01-rp-j3195648 cosmosDatabaseUrl:https://arohcpci01-rp-j3195648.documents.azure.com:443/ cpuRequest:630m csDeploymentStrategy:map[rollingUpdate:map[maxSurge:1 maxUnavailable:0] type:RollingUpdate] dataPlaneHAProxyImage:arohcpocpdev.azurecr.io/redhat-user-workloads/crt-redhat-acm-tenant/hypershift-shared-ingress-main@sha256:5e22710fe440869655329ecc0ef57b4ae15f4ef96e74a8904b1e6fafc0d87678 databaseAuthMethod:postgres databaseDisableTls:true databaseHost:ocm-cs-db databaseK8sSecretName:ocm-cs-db databaseName:ocm-cs-db databasePassword:TheBlurstOfTimes databasePort:5432 databaseServiceName:ocm-cs-db databaseUser:ocm databaseVolumeCapacity:512Mi debugMaxDuration:5m debugPort: defaultExpiration:0 deployment:map[zoneCount:3] environment:arohcpci01 forceMigration: fpaCertName:firstPartyCert2 gatewayUrl:http://127.0.0.1:9090 global:map[imageDigest:sha256:b6e44eb348113224aa292d06b6cbf4154e881fe0a8617e245e7c6555f046d495] imageDigest:sha256:b6e44eb348113224aa292d06b6cbf4154e881fe0a8617e245e7c6555f046d495 imageRegistry:arohcpsvcdev.azurecr.io imageRepository:app-sre/aro-hcp-clusters-service insecure:false jwksUrl:http://localhost logLevel:4 machinePoolMigrationWorkerPeriod:1h managedIdentitiesDataPlaneAudienceResource:https://dummy.org maximumExpiration:0 memoryLimit:unlimited memoryRequest:150Mi monitoringApiGroup:monitoring.coreos.com ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev ocpAcrResourceUrl:arohcpocpdev.azurecr.io oidcIssuerBaseUrl:https://arohcpci01oidcj3195648.z9.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpci01oidcj3195648.blob.core.windows.net/ postgresqlVersion:12 provisionShardClusterLimit:100 pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:app-sre/aro-hcp-clusters-service:pull workloadIdentityClientId:a5d5ff83-799c-44fd-a535-8c4386588d02 workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] region:canadacentral replicas:3 runtimeMode:aro-hcp serviceAccountName:clusters-service serviceKeyVaultName:aro-hcp-dev-svc-kv tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesEnabled:true tlsCertificatesIssuer:Self tokenUrl:http://localhost useAzureDB:false userDefinedDnsBaseDomain:i1.devshift.org]"
time=2026-10-28T17:01:13.550Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy
time=2026-10-28T17:01:13.717Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy namespace=clusters-service
time=2026-10-28T17:01:13.899Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy dryRun=false
time=2026-10-28T17:01:13.899Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy
time=2026-10-28T17:01:13.952Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy
time=2026-10-28T17:11:19.322Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="resource Deployment/clusters-service/clusters-service not ready. status: InProgress, message: Available: 2/3\ncontext deadline exceeded"
time=2026-10-28T17:11:19.322Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/clusters-service/clusters-service not ready. status: InProgress, message: Available: 2/3\ncontext deadline exceeded"
alert (1)
alert [svc] KubePodNotReady fired
16:20:01 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-28T17:07:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-64b56768b9-r7v9d", region="canadacentral", severity="warning"
  Description: Pod clusters-service/clusters-service-64b56768b9-r7v9d has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-28T17:07:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="fleet", pod="fleet-controller-7b9596cccb-g9thf", region="canadacentral", severity="warning"
  Description: Pod fleet/fleet-controller-7b9596cccb-g9thf has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-28T17:07:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="aro-hcp-admin-api", pod="admin-api-6d56468ffd-t9snj", region="canadacentral", severity="warning"
  Description: Pod aro-hcp-admin-api/admin-api-6d56468ffd-t9snj has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-28T17:07:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3195648-svc", component="kubernetes-infrastructure", namespace="aro-hcp-admin-api", pod="admin-api-6d56468ffd-rm24l", region="canadacentral", severity="warning"
  Description: Pod aro-hcp-admin-api/admin-api-6d56468ffd-rm24l has been in a non-ready state for longer than 5 minutes.
36m 16s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 0bb0b65
 centralus provision failed #7693 (open) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key vault "<url>"; provider Microsoft.ContainerService
16:19:05 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite templatize-pipeline
Show raw failure
time=2026-10-28T16:43:28.511Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T16:43:31.273Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T16:43:32.448Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=f309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2862080-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87
time=2026-10-28T16:45:32.811Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/f309630496c74d69129833a43742f2e9eec7ceb9d4411ea07b979bc73b89fd87/operationStatuses/08584084010736719502\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/cluster-rybldmoz73lrm\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j2862080-svc\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n                \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j2862080-1.vault.azure.net\\\\\\\"\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-28T16:47:32.817Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T16:47:34.143Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T16:47:35.068Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j2862080-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006
time=2026-10-28T16:49:05.326Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2862080-svc/providers/Microsoft.Resources/deployments/8fa925f61f96a8233f154e3da5c65d168518bb692c424a9f42720dad4bf60006/operationStatuses/08584084008309075646\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
2h 49m 42s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head dc65ef0
 canadacentral e2e failed ⚠#7777 (open) 3
e2e (3)
ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift
16:16:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x2b12a0180120>: 
    resource group "rg-candidate-4-22-9q8kddc4rfc6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-hscqkp" in resourcegroup="rg-candidate-4-22-9q8kddc4rfc6" to finish deleting: GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46e9ad9e-9f53-4038-ba36-f40dec0ff2b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46e9ad9e-9f53-4038-ba36-f40dec0ff2b1",
      "name": "46e9ad9e-9f53-4038-ba36-f40dec0ff2b1",
      "status": "Failed",
      "startTime": "2026-10-28T17:51:06.474152018Z",
      "endTime": "2026-10-28T18:15:11.708953381Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46e9ad9e-9f53-4038-ba36-f40dec0ff2b1
            correlation ID: 4741ead8-d72b-47f6-84e5-df7418434c6d
            client request ID: 9a224eab-3688-4c8c-9f47-6f8238e2c607
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x2b12a0180120>: 
    resource group "rg-candidate-4-22-9q8kddc4rfc6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-hscqkp" in resourcegroup="rg-candidate-4-22-9q8kddc4rfc6" to finish deleting: GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46e9ad9e-9f53-4038-ba36-f40dec0ff2b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46e9ad9e-9f53-4038-ba36-f40dec0ff2b1",
      "name": "46e9ad9e-9f53-4038-ba36-f40dec0ff2b1",
      "status": "Failed",
      "startTime": "2026-10-28T17:51:06.474152018Z",
      "endTime": "2026-10-28T18:15:11.708953381Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/46e9ad9e-9f53-4038-ba36-f40dec0ff2b1
            correlation ID: 4741ead8-d72b-47f6-84e5-df7418434c6d
            client request ID: 9a224eab-3688-4c8c-9f47-6f8238e2c607
occurred
ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift
16:16:33 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x17fd8fc22138>: 
    resource group "kusto-logs-2tq8m2fllmvs": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-2tq8m2fllmvs" to finish deleting: GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e027677d-2084-4178-b70c-b4e22dfc37b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e027677d-2084-4178-b70c-b4e22dfc37b1",
      "name": "e027677d-2084-4178-b70c-b4e22dfc37b1",
      "status": "Failed",
      "startTime": "2026-10-28T17:46:42.22634078Z",
      "endTime": "2026-10-28T18:10:50.94970388Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e027677d-2084-4178-b70c-b4e22dfc37b1
            correlation ID: fcf84780-ffda-49de-a03f-17dbeb0ef756
            client request ID: 070116fb-514a-47f7-ae2b-d06066429ea7
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x17fd8fc22138>: 
    resource group "kusto-logs-2tq8m2fllmvs": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-2tq8m2fllmvs" to finish deleting: GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e027677d-2084-4178-b70c-b4e22dfc37b1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e027677d-2084-4178-b70c-b4e22dfc37b1",
      "name": "e027677d-2084-4178-b70c-b4e22dfc37b1",
      "status": "Failed",
      "startTime": "2026-10-28T17:46:42.22634078Z",
      "endTime": "2026-10-28T18:10:50.94970388Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e027677d-2084-4178-b70c-b4e22dfc37b1
            correlation ID: fcf84780-ffda-49de-a03f-17dbeb0ef756
            client request ID: 070116fb-514a-47f7-ae2b-d06066429ea7
occurred
ERROR CODE: DeadlineExceeded; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift
16:16:33 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x2a43d7c880a8>: 
    resource group "private-kas-2t2bq848xclk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-2t2bq848xclk" to finish deleting: GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/827c464e-64e1-4e42-980f-59af842b0bed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/827c464e-64e1-4e42-980f-59af842b0bed",
      "name": "827c464e-64e1-4e42-980f-59af842b0bed",
      "status": "Failed",
      "startTime": "2026-10-28T17:45:46.805727964Z",
      "endTime": "2026-10-28T18:09:51.562148289Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/827c464e-64e1-4e42-980f-59af842b0bed
            correlation ID: c4e40049-9c6b-42d0-ae1f-bfdcd35c5399
            client request ID: 5bcb7ef3-7dfb-48a5-a7c7-360bec18ff93
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x2a43d7c880a8>: 
    resource group "private-kas-2t2bq848xclk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kas" in resourcegroup="private-kas-2t2bq848xclk" to finish deleting: GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/827c464e-64e1-4e42-980f-59af842b0bed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/827c464e-64e1-4e42-980f-59af842b0bed",
      "name": "827c464e-64e1-4e42-980f-59af842b0bed",
      "status": "Failed",
      "startTime": "2026-10-28T17:45:46.805727964Z",
      "endTime": "2026-10-28T18:09:51.562148289Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7963648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/827c464e-64e1-4e42-980f-59af842b0bed
            correlation ID: c4e40049-9c6b-42d0-ae1f-bfdcd35c5399
            client request ID: 5bcb7ef3-7dfb-48a5-a7c7-360bec18ff93
occurred
2h 6m 45s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 2a5af5f
 centralus e2e failed #7809 (open) 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want >= <count> (autoscaling min); node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
16:10:35 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x1c827a968340>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-9k6kks4sqmhh" to finish creating: GET https://rp.j5392000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e",
      "name": "e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e",
      "status": "Failed",
      "startTime": "2026-10-28T17:17:13.271950373Z",
      "endTime": "2026-10-28T17:36:19.733049446Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9k6kks4sqmhh--managed/providers/Microsoft.Compute/virtualMachines/x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9k6kks4sqmhh--managed/providers/Microsoft.Compute/virtualMachines/x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5392000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e
            correlation ID: e5b7a290-af02-4ff7-9f0a-851a1b652878
            client request ID: a046022e-1fb3-43ea-9318-d95c8087e3da
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0x1c827a968340>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-9k6kks4sqmhh" to finish creating: GET https://rp.j5392000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e",
      "name": "e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e",
      "status": "Failed",
      "startTime": "2026-10-28T17:17:13.271950373Z",
      "endTime": "2026-10-28T17:36:19.733049446Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 2, want \u003e= 3 (autoscaling min); node pool condition AllNodesHealthy is False: 1 of 3 machines are not healthy\nMachine x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9k6kks4sqmhh--managed/providers/Microsoft.Compute/virtualMachines/x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm to exist\n; node pool condition AllMachinesReady is False: 1 of 3 machines are not ready\nMachine x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-9k6kks4sqmhh--managed/providers/Microsoft.Compute/virtualMachines/x5b5y8h0f8t5k2t-autoscale-noaz-b47sx-574cm to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5392000.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e7e6b86c-a0ff-4c35-94e7-3aa8b285e19e
            correlation ID: e5b7a290-af02-4ff7-9f0a-851a1b652878
            client request ID: a046022e-1fb3-43ea-9318-d95c8087e3da
occurred
2h 20m 20s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 3903554 · merge 3903554
 canadacentral n/a passed #7786 (merged) 0 n/a
2h 20m 2s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 1e655fa
 centralus e2e failed ⚠#7757 (open) 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '1' s... message truncated; provider Microsoft.RedHatOpenShift
15:54:37 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-8844z2" for cluster "cluster-candidate-4-22-8844z2"
Unexpected error:
    <*fmt.wrapError | 0x34a8b7c1a9e0>: 
    failed to create NodePool np-8844z2: failed waiting for nodepool="np-8844z2" for cluster "cluster-candidate-4-22-8844z2" in resourcegroup="rg-candidate-4-22-p8n8vcw2knj6" to finish creating: GET https://rp.j7668608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e163bbab-c728-4e64-85d2-39934825ad2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e163bbab-c728-4e64-85d2-39934825ad2c",
      "name": "e163bbab-c728-4e64-85d2-39934825ad2c",
      "status": "Failed",
      "startTime": "2026-10-28T17:07:17.250607245Z",
      "endTime": "2026-10-28T17:26:25.565644744Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y8h5w3s1s0z4u9d-np-8844z2-q5kw2-zpqqj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-p8n8vcw2knj6-candidate-8844z2--managed/providers/Microsoft.Compute/virtualMachines/y8h5w3s1s0z4u9d-np-8844z2-q5kw2-zpqqj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y8h5w3s1s0z4u9d-np-8844z2-q5kw2-qsnbp: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-22-p8n8vcw2knj6-candidate-8844z2--managed/y8h5w3s1s0z4u9d-np-8844z2-q5kw2-qsnbp-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-p8n8vcw2knj6-candidate-8844z2--managed/providers/Microsoft.Network/networkInterfaces/y8h5w3s1s0z4u9d-np-8844z2-q5kw2-qsnbp-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '1' s... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7668608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e163bbab-c728-4e64-85d2-39934825ad2c
            correlation ID: 8a8572c9-5da8-4651-8233-b7c79a67a3b3
            client request ID: fc96c7e7-a6f5-442a-9e18-6d3eec0c11ee
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:220]: failed to create node pool "np-8844z2" for cluster "cluster-candidate-4-22-8844z2"
Unexpected error:
    <*fmt.wrapError | 0x34a8b7c1a9e0>: 
    failed to create NodePool np-8844z2: failed waiting for nodepool="np-8844z2" for cluster "cluster-candidate-4-22-8844z2" in resourcegroup="rg-candidate-4-22-p8n8vcw2knj6" to finish creating: GET https://rp.j7668608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e163bbab-c728-4e64-85d2-39934825ad2c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e163bbab-c728-4e64-85d2-39934825ad2c",
      "name": "e163bbab-c728-4e64-85d2-39934825ad2c",
      "status": "Failed",
      "startTime": "2026-10-28T17:07:17.250607245Z",
      "endTime": "2026-10-28T17:26:25.565644744Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y8h5w3s1s0z4u9d-np-8844z2-q5kw2-zpqqj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-p8n8vcw2knj6-candidate-8844z2--managed/providers/Microsoft.Compute/virtualMachines/y8h5w3s1s0z4u9d-np-8844z2-q5kw2-zpqqj to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine y8h5w3s1s0z4u9d-np-8844z2-q5kw2-qsnbp: Failed: interfaces failed to create or update. err: failed to get existing resource rg-candidate-4-22-p8n8vcw2knj6-candidate-8844z2--managed/y8h5w3s1s0z4u9d-np-8844z2-q5kw2-qsnbp-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-p8n8vcw2knj6-candidate-8844z2--managed/providers/Microsoft.Network/networkInterfaces/y8h5w3s1s0z4u9d-np-8844z2-q5kw2-qsnbp-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'e92b9f76-b040-4cf6-a4dd-5c8bdc759e69' exceeded. Please try again after '1' s... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j7668608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e163bbab-c728-4e64-85d2-39934825ad2c
            correlation ID: 8a8572c9-5da8-4651-8233-b7c79a67a3b3
            client request ID: fc96c7e7-a6f5-442a-9e18-6d3eec0c11ee
occurred
2h 17m 56s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 7170922
 centralus e2e failed #7695 (open) 1
e2e (1)
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
15:35:19 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x2db4fc89320>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zbnm2gmmrc7k" to finish creating: GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b",
      "name": "13b873cd-2efa-4559-ab15-ff37ab2d031b",
      "status": "Failed",
      "startTime": "2026-10-28T16:52:57.688694522Z",
      "endTime": "2026-10-28T17:12:06.752450543Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
            correlation ID: 596c64f1-beb7-4366-a3b9-24ac7fe089dc
            client request ID: 7b2dfc27-0f04-4e6d-a8a1-787b24c6e549
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x2db4fc89320>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zbnm2gmmrc7k" to finish creating: GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b",
      "name": "13b873cd-2efa-4559-ab15-ff37ab2d031b",
      "status": "Failed",
      "startTime": "2026-10-28T16:52:57.688694522Z",
      "endTime": "2026-10-28T17:12:06.752450543Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NodeNotHealthy: * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j8c0h2z2t3o8e9b-np-delete-main-69xnv-msqbr: NotReady: * NodeHealthy:\n  * Node.Ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8443648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b873cd-2efa-4559-ab15-ff37ab2d031b
            correlation ID: 596c64f1-beb7-4366-a3b9-24ac7fe089dc
            client request ID: 7b2dfc27-0f04-4e6d-a8a1-787b24c6e549
11m 54s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head b79a3e3
 n/a unknown failed #6951 (open) 1 non-artifact-backed failure (no junit artifacts)
2h 30m 47s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head bb1de19 · merge bb1de19
 canadacentral n/a passed #7776 (merged) 0 n/a
2h 25m 15s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head e7c23ba
 centralus e2e failed #7704 (open) 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift
15:15:04 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x335ced2f1b40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-kfwkzrbn6vbm" to finish creating: GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb",
      "name": "77628145-9ee8-4d12-9dd2-5dced29b5ecb",
      "status": "Failed",
      "startTime": "2026-10-28T16:28:12.935779531Z",
      "endTime": "2026-10-28T16:47:20.605052936Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7: Failed: virtualmachine failed to create or update. err: failed to get existing resource fips-enabled-kfwkzrbn6vbm--managed/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-kfwkzrbn6vbm--managed/providers/Microsoft.Compute/virtualMachines/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https:... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
            correlation ID: c83d59d4-8297-41d1-87c1-fbe0352a6163
            client request ID: 4dcc2ce6-6521-4d7f-b86f-f0309a80ecee
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x335ced2f1b40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-kfwkzrbn6vbm" to finish creating: GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb",
      "name": "77628145-9ee8-4d12-9dd2-5dced29b5ecb",
      "status": "Failed",
      "startTime": "2026-10-28T16:28:12.935779531Z",
      "endTime": "2026-10-28T16:47:20.605052936Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7: Failed: virtualmachine failed to create or update. err: failed to get existing resource fips-enabled-kfwkzrbn6vbm--managed/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7 (service: virtualmachine): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-kfwkzrbn6vbm--managed/providers/Microsoft.Compute/virtualMachines/j6k4t1t6z4j2k5z-np-1-5nm58-fsnz7\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'b117cf0f-0276-4486-9691-4f00f5a90e1b' exceeded. Please try again after '1' seconds after additional tokens are available. Refer to https:... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4206336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77628145-9ee8-4d12-9dd2-5dced29b5ecb
            correlation ID: c83d59d4-8297-41d1-87c1-fbe0352a6163
            client request ID: 4dcc2ce6-6521-4d7f-b86f-f0309a80ecee
occurred
2h 31m 38s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 4ec1091 · merge 4ec1091
 canadacentral n/a passed #7782 (merged) 0 n/a
27m 41s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head e2aa104 · merge e2aa104
 n/a unknown failed #7705 (merged) 1 non-artifact-backed failure (no junit artifacts)
2h 58m 39s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head c17975b
 westus3 e2e failed ★#7802 (open) 1
e2e (1)
[events] e2e-csi-disk-<id> PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node <node> (reason=ProvisioningFailed, count=8, age=7m53s)
15:14:57 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-cptgtj"
Unexpected error:
    <*errors.joinError | 0x199ff3994258>: 
    VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.006s: pod e2e-csi-disk-s4t9m/e2e-csi-pod phase is "Pending", expected Succeeded; PVC is Bound
    [storageclass] name=managed-csi provisioner=disk.csi.azure.com
    [pvc] e2e-csi-disk-s4t9m/e2e-csi-pvc phase=Bound volumeName=pvc-742751d0-4b4e-4c69-b7d6-859886d48626 storageClass=managed-csi
    [pod] e2e-csi-disk-s4t9m/e2e-csi-pod phase=Pending reason= message=
    [pod] condition type=PodReadyToStartContainers status=False reason= message=
    [pod] condition type=Initialized status=True reason= message=
    [pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=PodScheduled status=True reason= message=
    [pod] container volume-check ready=false restartCount=0
    [pod]   waiting reason=ContainerCreating message=
    [events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-s4t9m/e2e-csi-pod to l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=Scheduled, count=0, age=5m24s)
    [events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: AttachVolume.Attach failed for volume "pvc-742751d0-4b4e-4c69-b7d6-859886d48626" : rpc error: code = Internal desc = Attach volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 to instance l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 failed with throttled due to too many requests (reason=FailedAttachVolume, count=9, age=2m0s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=19, age=5m33s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-s4t9m/e2e-csi-pvc" (reason=Provisioning, count=9, age=5m45s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=ProvisioningFailed, count=8, age=7m53s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 (reason=ProvisioningSucceeded, count=1, age=5m24s)
    {
        errs: [
            <*fmt.wrapError | 0x199ff3d7a160>{
                msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.006s: pod e2e-csi-disk-s4t9m/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-s4t9m/e2e-csi-pvc phase=Bound volumeName=pvc-742751d0-4b4e-4c69-b7d6-859886d48626 storageClass=managed-csi\n[pod] e2e-csi-disk-s4t9m/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-s4t9m/e2e-csi-pod to l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=Scheduled, count=0, age=5m24s)\n[events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: AttachVolume.Attach failed for volume \"pvc-742751d0-4b4e-4c69-b7d6-859886d48626\" : rpc error: code = Internal desc = Attach volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 to instance l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 failed with throttled due to too many requests (reason=FailedAttachVolume, count=9, age=2m0s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=19, age=5m33s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-s4t9m/e2e-csi-pvc\" (reason=Provisioning, count=9, age=5m45s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=ProvisioningFailed, count=8, age=7m53s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 (reason=ProvisioningSucceeded, count=1, age=5m24s)",
                err: <*fmt.wrapError | 0x199ff3d7a020>{
                    msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.006s: pod e2e-csi-disk-s4t9m/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-s4t9m/e2e-csi-pvc phase=Bound volumeName=pvc-742751d0-4b4e-4c69-b7d6-859886d48626 storageClass=managed-csi\n[pod] e2e-csi-disk-s4t9m/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-s4t9m/e2e-csi-pod to l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=Scheduled, count=0, age=5m24s)\n[events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: AttachVolume.Attach failed for volume \"pvc-742751d0-4b4e-4c69-b7d6-859886d48626\" : rpc error: code = Internal desc = Attach volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 to instance l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 failed with throttled due to too many requests (reason=FailedAttachVolume, count=9, age=2m0s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=19, age=5m33s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-s4t9m/e2e-csi-pvc\" (reason=Provisioning, count=9, age=5m45s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=ProvisioningFailed, count=8, age=7m53s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 (reason=ProvisioningSucceeded, count=1, age=5m24s)",
                    err: <*verifiers.pollObservationError | 0x199ff3d7a680>{
                        observation: <*errors.errorString | 0x199ff36faf30>{
                            s: "pod e2e-csi-disk-s4t9m/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound",
                        },
                        interruption: <*fmt.wrapError | 0x199ff3d7a660>{
                            msg: "failed to get PVC e2e-csi-disk-s4t9m/e2e-csi-pvc: context deadline exceeded",
                            err: <context.deadlineExceededError>{},
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-cptgtj"
Unexpected error:
    <*errors.joinError | 0x199ff3994258>: 
    VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.006s: pod e2e-csi-disk-s4t9m/e2e-csi-pod phase is "Pending", expected Succeeded; PVC is Bound
    [storageclass] name=managed-csi provisioner=disk.csi.azure.com
    [pvc] e2e-csi-disk-s4t9m/e2e-csi-pvc phase=Bound volumeName=pvc-742751d0-4b4e-4c69-b7d6-859886d48626 storageClass=managed-csi
    [pod] e2e-csi-disk-s4t9m/e2e-csi-pod phase=Pending reason= message=
    [pod] condition type=PodReadyToStartContainers status=False reason= message=
    [pod] condition type=Initialized status=True reason= message=
    [pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=PodScheduled status=True reason= message=
    [pod] container volume-check ready=false restartCount=0
    [pod]   waiting reason=ContainerCreating message=
    [events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-s4t9m/e2e-csi-pod to l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=Scheduled, count=0, age=5m24s)
    [events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: AttachVolume.Attach failed for volume "pvc-742751d0-4b4e-4c69-b7d6-859886d48626" : rpc error: code = Internal desc = Attach volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 to instance l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 failed with throttled due to too many requests (reason=FailedAttachVolume, count=9, age=2m0s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=19, age=5m33s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-s4t9m/e2e-csi-pvc" (reason=Provisioning, count=9, age=5m45s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=ProvisioningFailed, count=8, age=7m53s)
    [events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 (reason=ProvisioningSucceeded, count=1, age=5m24s)
    {
        errs: [
            <*fmt.wrapError | 0x199ff3d7a160>{
                msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.006s: pod e2e-csi-disk-s4t9m/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-s4t9m/e2e-csi-pvc phase=Bound volumeName=pvc-742751d0-4b4e-4c69-b7d6-859886d48626 storageClass=managed-csi\n[pod] e2e-csi-disk-s4t9m/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-s4t9m/e2e-csi-pod to l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=Scheduled, count=0, age=5m24s)\n[events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: AttachVolume.Attach failed for volume \"pvc-742751d0-4b4e-4c69-b7d6-859886d48626\" : rpc error: code = Internal desc = Attach volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 to instance l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 failed with throttled due to too many requests (reason=FailedAttachVolume, count=9, age=2m0s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=19, age=5m33s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-s4t9m/e2e-csi-pvc\" (reason=Provisioning, count=9, age=5m45s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=ProvisioningFailed, count=8, age=7m53s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 (reason=ProvisioningSucceeded, count=1, age=5m24s)",
                err: <*fmt.wrapError | 0x199ff3d7a020>{
                    msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.006s: pod e2e-csi-disk-s4t9m/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-s4t9m/e2e-csi-pvc phase=Bound volumeName=pvc-742751d0-4b4e-4c69-b7d6-859886d48626 storageClass=managed-csi\n[pod] e2e-csi-disk-s4t9m/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-s4t9m/e2e-csi-pod to l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=Scheduled, count=0, age=5m24s)\n[events] e2e-csi-disk-s4t9m Pod/e2e-csi-pod: AttachVolume.Attach failed for volume \"pvc-742751d0-4b4e-4c69-b7d6-859886d48626\" : rpc error: code = Internal desc = Attach volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 to instance l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 failed with throttled due to too many requests (reason=FailedAttachVolume, count=9, age=2m0s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: waiting for first consumer to be created before binding (reason=WaitForFirstConsumer, count=1, age=10m0s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=19, age=5m33s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-s4t9m/e2e-csi-pvc\" (reason=Provisioning, count=9, age=5m45s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node l1m3f0i9q3m4q6d-np-cptgtj-w9vkt-7rxb5 (reason=ProvisioningFailed, count=8, age=7m53s)\n[events] e2e-csi-disk-s4t9m PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-742751d0-4b4e-4c69-b7d6-859886d48626 (reason=ProvisioningSucceeded, count=1, age=5m24s)",
                    err: <*verifiers.pollObservationError | 0x199ff3d7a680>{
                        observation: <*errors.errorString | 0x199ff36faf30>{
                            s: "pod e2e-csi-disk-s4t9m/e2e-csi-pod phase is \"Pending\", expected Succeeded; PVC is Bound",
                        },
                        interruption: <*fmt.wrapError | 0x199ff3d7a660>{
                            msg: "failed to get PVC e2e-csi-disk-s4t9m/e2e-csi-pvc: context deadline exceeded",
                            err: <context.deadlineExceededError>{},
                        },
                    },
                },
            },
        ],
    }
occurred
2h 16m 39s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head a3c518a · merge a3c518a
 westus3 n/a passed #7783 (merged) 0 n/a
1h 6m 28s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 6f4a3de
 canadacentral provision failed #7781 (open) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code AzureKeyVaultKmsValidateIdentityPermissionCustomerError; detail message The identity does not have keys encrypt/decrypt permission on key vault "<url>"; provider Microsoft.ContainerService
15:05:59 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite templatize-pipeline
Show raw failure
time=2026-10-28T15:32:48.082Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T15:32:50.394Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T15:32:51.656Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=51bb59b51713025e67da1bbb1f81509d9f0e687a1b531b2b697347d81ab333d6 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1862528-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F51bb59b51713025e67da1bbb1f81509d9f0e687a1b531b2b697347d81ab333d6
time=2026-10-28T15:35:22.185Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster step=cluster error="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1862528-svc/providers/Microsoft.Resources/deployments/51bb59b51713025e67da1bbb1f81509d9f0e687a1b531b2b697347d81ab333d6/operationStatuses/08584084053144753487\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1862528-svc/providers/Microsoft.Resources/deployments/cluster-5xxgmkpcjnpl2\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1862528-svc/providers/Microsoft.ContainerService/managedClusters/ci01-j1862528-svc\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"AzureKeyVaultKmsValidateIdentityPermissionCustomerError\\\",\\r\\n                \\\"message\\\": \\\"The identity does not have keys encrypt/decrypt permission on key vault \\\\\\\"https://ah-ci01-se-j1862528-1.vault.azure.net\\\\\\\"\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-28T15:37:22.187Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-28T15:37:23.185Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-28T15:37:24.435Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=f330c62a337aea146d3097645390c7352ebc193053cfec3eadc02b161a2aba5f portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1862528-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ff330c62a337aea146d3097645390c7352ebc193053cfec3eadc02b161a2aba5f
time=2026-10-28T15:38:54.741Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1862528-svc/providers/Microsoft.Resources/deployments/f330c62a337aea146d3097645390c7352ebc193053cfec3eadc02b161a2aba5f/operationStatuses/08584084050417029665\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"Failed to obtain the resource body, because the resource is currently in a 'Failed' provisioning state. If this resource has not been modified by a prior step in this deployment, you may need to address this before retrying the deployment.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
14m 46s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head d21a3fd
 n/a unknown failed #7780 (open) 1 non-artifact-backed failure (no junit artifacts)
1m 50s pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 6de3a51 · merge bb1de19
 n/a unknown failed #7776 (merged) 1 non-artifact-backed failure (no junit artifacts)
24m 52s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 3903554 · merge 3903554
 n/a unknown failed #7786 (merged) 1 non-artifact-backed failure (no junit artifacts)
25m 12s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 608bb71
 centralus alert failed #7793 (open) 3
alert (3)
ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement
14:30:04 UTC · failed at alert · test [aro-hcp-observability] [svc] alert collection is complete · suite aro-hcp-tests
Show raw failure
alert collection incomplete
failed to fetch svc alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8739840/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j8739840' could not be found."
  }
}
--------------------------------------------------------------------------------
ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement
14:30:04 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert collection is complete · suite aro-hcp-tests
Show raw failure
alert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8739840/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j8739840' could not be found."
  }
}
--------------------------------------------------------------------------------
ERROR CODE: ResourceGroupNotFound; provider Microsoft.Insights
14:30:04 UTC · failed at alert · test [aro-hcp-observability] [infra] alert collection is complete · suite aro-hcp-tests
Show raw failure
alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8739840: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j8739840/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j8739840' could not be found."
  }
}
--------------------------------------------------------------------------------
50m 0s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head a3c518a · merge a3c518a
 canadacentral provision failed ★#7783 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code AssociationFailedAtResourceProvider; detail message Association provisioning failed at resource provider; provider Microsoft.Network
13:13:49 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/nsp (stamp=2) · suite templatize-pipeline
Show raw failure
time=2026-10-28T13:56:43.120Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 description="Step nsp\n  Kind: ARM\n  Template: templates/mgmt-nsp.bicep\n  Parameters: configurations/mgmt-nsp.tmpl.bicepparam"
time=2026-10-28T13:56:43.546Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2
time=2026-10-28T13:56:43.936Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 deployment=27369930d49e6b87bdd6e0a76bb435d03ceef9149d6c362d06dfe1d47f9f5244 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j7323392-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F27369930d49e6b87bdd6e0a76bb435d03ceef9149d6c362d06dfe1d47f9f5244
time=2026-10-28T13:58:14.248Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7323392-mgmt-2/providers/Microsoft.Resources/deployments/27369930d49e6b87bdd6e0a76bb435d03ceef9149d6c362d06dfe1d47f9f5244/operationStatuses/08584084110817295270\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7323392-mgmt-2/providers/Microsoft.Resources/deployments/profile-sxizw5tprl6fg\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7323392-mgmt-2/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j7323392-mgmt-2/resourceAssociations/nsp-j7323392-mgmt-2-ra-4bf3dc7e-4183-5ff8-b0f7-92d18b1331a1\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"AssociationFailedAtResourceProvider\\\",\\r\\n                \\\"message\\\": \\\"Association provisioning failed at resource provider\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
13m 13s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 5a9d9b5
 n/a unknown failed #7793 (open) 1 non-artifact-backed failure (no junit artifacts)
2h 27m 36s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head de22b69 · merge de22b69
 centralus n/a passed #6718 (merged) 0 n/a
2h 13m 16s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 157b5da
 westus3 alert, e2e failed ⚠#7662 (open) 3
e2e (2)
missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11333, acm-agent=0
12:14:35 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.005s.
Expected success, but got an error:
    <*errors.errorString | 0xca1fe06c6d0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11333, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11333, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.005s.
Expected success, but got an error:
    <*errors.errorString | 0xca1fe06c6d0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11333, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=11333, acm-agent=0",
    }
VerifyAllAPIServicesAvailable failed: apiservice/v1beta1
12:14:35 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 5.1 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:183]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0xc83633402b8>: 
    VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        errs: [
            <*fmt.wrapError | 0xc83630bcee0>{
                msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                err: <*errors.joinError | 0xc8362e9fb78>{
                    errs: [
                        <*errors.errorString | 0xc83633b5610>{
                            s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        },
                    ],
                },
            },
        ],
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:183]: cluster viability check failed
Expected success, but got an error:
    <*errors.joinError | 0xc83633402b8>: 
    VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get "https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1": tls: failed to verify certificate: x509: certificate signed by unknown authority
    {
        errs: [
            <*fmt.wrapError | 0xc83630bcee0>{
                msg: "VerifyAllAPIServicesAvailable failed: apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                err: <*errors.joinError | 0xc8362e9fb78>{
                    errs: [
                        <*errors.errorString | 0xc83633b5610>{
                            s: "apiservice/v1beta1.metrics.k8s.io is not available because: failing or missing response from https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1: Get \"https://10.128.0.29:10250/apis/metrics.k8s.io/v1beta1\": tls: failed to verify certificate: x509: certificate signed by unknown authority",
                        },
                    ],
                },
            },
        ],
    }
alert (1)
alert [svc] KubePodCrashLooping fired
12:14:35 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubePodCrashLooping does not fire · suite aro-hcp-tests
Show raw failure
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-28T13:27:25Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0229248-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.125:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0229248/providers/microsoft.monitor/accounts/services-j0229248", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-phdkm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5471b0ac-6a29-48cb-ab86-2893cd772fbd"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-phdkm (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-28T13:29:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0229248-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.195:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0229248/providers/microsoft.monitor/accounts/services-j0229248", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-84g47", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9462daaa-47d3-4b5e-b73b-01a9f857a137"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-84g47 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-28T13:29:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0229248-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.195:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0229248/providers/microsoft.monitor/accounts/services-j0229248", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-84g47", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9462daaa-47d3-4b5e-b73b-01a9f857a137"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-84g47 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-28T13:32:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0229248-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.125:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0229248/providers/microsoft.monitor/accounts/services-j0229248", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-phdkm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5471b0ac-6a29-48cb-ab86-2893cd772fbd"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-phdkm (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
2h 14m 55s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head c17975b
 centralus e2e failed ★#7802 (open) 1
e2e (1)
error code: CANCEL
11:46:29 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-th8csc"
Unexpected error:
    <*errors.joinError | 0x28585f5aa1e0>: 
    VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-5zpk7/e2e-csi-pvc phase is "Pending", expected Bound; pod phase is "Pending"
    [storageclass] name=managed-csi provisioner=disk.csi.azure.com
    [pvc] e2e-csi-disk-5zpk7/e2e-csi-pvc phase=Bound volumeName=pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f storageClass=managed-csi
    [pod] e2e-csi-disk-5zpk7/e2e-csi-pod phase=Pending reason= message=
    [pod] condition type=PodReadyToStartContainers status=False reason= message=
    [pod] condition type=Initialized status=True reason= message=
    [pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=PodScheduled status=True reason= message=
    [pod] container volume-check ready=false restartCount=0
    [pod]   waiting reason=ContainerCreating message=
    [events] e2e-csi-disk-5zpk7 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-5zpk7/e2e-csi-pod to t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=Scheduled, count=0, age=12s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-5zpk7/e2e-csi-pvc" (reason=Provisioning, count=10, age=36s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=ProvisioningFailed, count=5, age=9m33s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m48s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=8m47s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=3m33s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f (reason=ProvisioningSucceeded, count=1, age=13s)
    {
        errs: [
            <*fmt.wrapError | 0x28585f462380>{
                msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-5zpk7/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-5zpk7/e2e-csi-pvc phase=Bound volumeName=pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f storageClass=managed-csi\n[pod] e2e-csi-disk-5zpk7/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-5zpk7 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-5zpk7/e2e-csi-pod to t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=Scheduled, count=0, age=12s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-5zpk7/e2e-csi-pvc\" (reason=Provisioning, count=10, age=36s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=ProvisioningFailed, count=5, age=9m33s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m48s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=8m47s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=3m33s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f (reason=ProvisioningSucceeded, count=1, age=13s)",
                err: <*fmt.wrapError | 0x28585e8a4800>{
                    msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-5zpk7/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-5zpk7/e2e-csi-pvc phase=Bound volumeName=pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f storageClass=managed-csi\n[pod] e2e-csi-disk-5zpk7/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-5zpk7 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-5zpk7/e2e-csi-pod to t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=Scheduled, count=0, age=12s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-5zpk7/e2e-csi-pvc\" (reason=Provisioning, count=10, age=36s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=ProvisioningFailed, count=5, age=9m33s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m48s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=8m47s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=3m33s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f (reason=ProvisioningSucceeded, count=1, age=13s)",
                    err: <*verifiers.pollObservationError | 0x28585e8a43c0>{
                        observation: <*errors.errorString | 0x28585ee0a670>{
                            s: "PVC e2e-csi-disk-5zpk7/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"",
                        },
                        interruption: <*fmt.wrapError | 0x28585e8a4320>{
                            msg: "failed to get PVC e2e-csi-disk-5zpk7/e2e-csi-pvc: context deadline exceeded",
                            err: <context.deadlineExceededError>{},
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-th8csc"
Unexpected error:
    <*errors.joinError | 0x28585f5aa1e0>: 
    VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-5zpk7/e2e-csi-pvc phase is "Pending", expected Bound; pod phase is "Pending"
    [storageclass] name=managed-csi provisioner=disk.csi.azure.com
    [pvc] e2e-csi-disk-5zpk7/e2e-csi-pvc phase=Bound volumeName=pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f storageClass=managed-csi
    [pod] e2e-csi-disk-5zpk7/e2e-csi-pod phase=Pending reason= message=
    [pod] condition type=PodReadyToStartContainers status=False reason= message=
    [pod] condition type=Initialized status=True reason= message=
    [pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]
    [pod] condition type=PodScheduled status=True reason= message=
    [pod] container volume-check ready=false restartCount=0
    [pod]   waiting reason=ContainerCreating message=
    [events] e2e-csi-disk-5zpk7 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-5zpk7/e2e-csi-pod to t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=Scheduled, count=0, age=12s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim "e2e-csi-disk-5zpk7/e2e-csi-pvc" (reason=Provisioning, count=10, age=36s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=ProvisioningFailed, count=5, age=9m33s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m48s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=8m47s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=3m33s)
    [events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f (reason=ProvisioningSucceeded, count=1, age=13s)
    {
        errs: [
            <*fmt.wrapError | 0x28585f462380>{
                msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) failed: VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-5zpk7/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-5zpk7/e2e-csi-pvc phase=Bound volumeName=pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f storageClass=managed-csi\n[pod] e2e-csi-disk-5zpk7/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-5zpk7 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-5zpk7/e2e-csi-pod to t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=Scheduled, count=0, age=12s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-5zpk7/e2e-csi-pvc\" (reason=Provisioning, count=10, age=36s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=ProvisioningFailed, count=5, age=9m33s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m48s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=8m47s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=3m33s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f (reason=ProvisioningSucceeded, count=1, age=13s)",
                err: <*fmt.wrapError | 0x28585e8a4800>{
                    msg: "VerifyCSIPersistentVolume(disk.csi.azure.com) timed out after 10m0.004s: PVC e2e-csi-disk-5zpk7/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"\n[storageclass] name=managed-csi provisioner=disk.csi.azure.com\n[pvc] e2e-csi-disk-5zpk7/e2e-csi-pvc phase=Bound volumeName=pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f storageClass=managed-csi\n[pod] e2e-csi-disk-5zpk7/e2e-csi-pod phase=Pending reason= message=\n[pod] condition type=PodReadyToStartContainers status=False reason= message=\n[pod] condition type=Initialized status=True reason= message=\n[pod] condition type=Ready status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=ContainersReady status=False reason=ContainersNotReady message=containers with unready status: [volume-check]\n[pod] condition type=PodScheduled status=True reason= message=\n[pod] container volume-check ready=false restartCount=0\n[pod]   waiting reason=ContainerCreating message=\n[events] e2e-csi-disk-5zpk7 Pod/e2e-csi-pod: Successfully assigned e2e-csi-disk-5zpk7/e2e-csi-pod to t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=Scheduled, count=0, age=12s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: External provisioner is provisioning volume for claim \"e2e-csi-disk-5zpk7/e2e-csi-pvc\" (reason=Provisioning, count=10, age=36s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: error generating accessibility requirements: no topology key found for node t8z9t1e8w4t1q5i-np-th8csc-h6n6d-49dfg (reason=ProvisioningFailed, count=5, age=9m33s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Waiting for a volume to be created either by the external provisioner 'disk.csi.azure.com' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered. (reason=ExternalProvisioning, count=26, age=3m48s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = stream terminated by RST_STREAM with error code: CANCEL (reason=ProvisioningFailed, count=1, age=8m47s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: rpc error: code = DeadlineExceeded desc = context deadline exceeded (reason=ProvisioningFailed, count=3, age=3m33s)\n[events] e2e-csi-disk-5zpk7 PersistentVolumeClaim/e2e-csi-pvc: Successfully provisioned volume pvc-ec65a1b9-74c2-42c0-961c-c9b41647402f (reason=ProvisioningSucceeded, count=1, age=13s)",
                    err: <*verifiers.pollObservationError | 0x28585e8a43c0>{
                        observation: <*errors.errorString | 0x28585ee0a670>{
                            s: "PVC e2e-csi-disk-5zpk7/e2e-csi-pvc phase is \"Pending\", expected Bound; pod phase is \"Pending\"",
                        },
                        interruption: <*fmt.wrapError | 0x28585e8a4320>{
                            msg: "failed to get PVC e2e-csi-disk-5zpk7/e2e-csi-pvc: context deadline exceeded",
                            err: <context.deadlineExceededError>{},
                        },
                    },
                },
            },
        ],
    }
occurred
2h 12m 31s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 4ec1091 · merge 4ec1091
 canadacentral alert, e2e failed #7782 (merged) 5
e2e (1)
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
11:45:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-llxcnv" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x1b596ab10140>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-llxcnv: failed waiting for cluster="cp-ystream-upgrade-4-22-llxcnv" in resourcegroup="rg-cp-ystream-upgrade-4-22-468dv2nskdkq" to finish creating: GET https://rp.j6945280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e79580c0-76a4-4e28-820e-080632c7ada2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e79580c0-76a4-4e28-820e-080632c7ada2",
      "name": "e79580c0-76a4-4e28-820e-080632c7ada2",
      "status": "Failed",
      "startTime": "2026-10-28T12:38:01.23060152Z",
      "endTime": "2026-10-28T13:07:03.13964352Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6945280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e79580c0-76a4-4e28-820e-080632c7ada2
            correlation ID: 8869047d-1b08-4ff0-8212-00001a9147bd
            client request ID: 0bcb310d-278e-4828-86a5-41d1c5537311
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-llxcnv" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x1b596ab10140>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-llxcnv: failed waiting for cluster="cp-ystream-upgrade-4-22-llxcnv" in resourcegroup="rg-cp-ystream-upgrade-4-22-468dv2nskdkq" to finish creating: GET https://rp.j6945280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e79580c0-76a4-4e28-820e-080632c7ada2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e79580c0-76a4-4e28-820e-080632c7ada2",
      "name": "e79580c0-76a4-4e28-820e-080632c7ada2",
      "status": "Failed",
      "startTime": "2026-10-28T12:38:01.23060152Z",
      "endTime": "2026-10-28T13:07:03.13964352Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6945280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e79580c0-76a4-4e28-820e-080632c7ada2
            correlation ID: 8869047d-1b08-4ff0-8212-00001a9147bd
            client request ID: 0bcb310d-278e-4828-86a5-41d1c5537311
occurred
alert (4)
alert [hcp] KubeDeploymentRolloutStuck fired
11:45:41 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentRolloutStuck does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T13:22:31Z
  Ended:       2026-10-28T13:28:32Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-fct9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/capi-provider is not progressing for longer than 30 minutes.
alert [hcp] KubePodCrashLooping fired
11:45:41 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T12:59:31Z
  Ended:       2026-10-28T13:28:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5496ebfd-2ab4-43b8-9f9a-e0483581f4bd"
  Description: Pod ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-28T13:00:33Z
  Ended:       2026-10-28T13:27:30Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5496ebfd-2ab4-43b8-9f9a-e0483581f4bd"
  Description: Pod ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
alert [hcp] KubeDeploymentReplicasMismatch fired
11:45:41 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T13:12:32Z
  Ended:       2026-10-28T13:28:32Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-fct9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert [hcp] KubeStatefulSetReplicasMismatch fired
11:45:41 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T12:57:34Z
  Ended:       2026-10-28T13:23:33Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-fct9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-28T12:57:34Z
  Ended:       2026-10-28T13:23:33Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j6945280-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.108:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6945280/providers/microsoft.monitor/accounts/hcps-j6945280", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-fct9q", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tnptam9khdl3svuvm4seuq5gr0ieit6-v5j4g5d3s9p2z6k/etcd has not matched the expected number of replicas for longer than 15 minutes.
2h 11m 34s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head db216ea · merge db216ea
 westus3 n/a passed #7800 (merged) 0 n/a
2h 34m 58s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 659a47e · merge 659a47e
 westus3 n/a passed #7753 (merged) 0 n/a
2h 43m 15s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 3903554 · merge 3903554
 westus3 e2e failed ★#7786 (merged) 3
e2e (3)
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-swrng7' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network
11:14:07 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-v9km2q"
Unexpected error:
    <*errors.joinError | 0x1f131b624198>: 
    VerifyIngressDNSRecord(aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud) timed out after 10m0.003s: DNS A record "*.e2e-dns-swrng7" not yet present in zone rg-candidate-4-23-2f2k8zzmfdnf-candidate-v9km2q--managed/aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2f2k8zzmfdnf-candidate-v9km2q--managed/providers/Microsoft.Network/dnsZones/aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud/A/*.e2e-dns-swrng7
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-swrng7' does not exist in resource group 'rg-candidate-4-23-2f2k8zzmfdnf-candidate-v9km2q--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2f2k8zzmfdnf-candidate-v9km2q--managed/providers/Microsoft.Network/dnsZones/aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud/A/*.e2e-dns-swrng7
            correlation ID: f5ba2020-eed4-41d4-be4e-2d7ff36da168
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-23-v9km2q"
Unexpected error:
    <*errors.joinError | 0x1f131b624198>: 
    VerifyIngressDNSRecord(aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud) timed out after 10m0.003s: DNS A record "*.e2e-dns-swrng7" not yet present in zone rg-candidate-4-23-2f2k8zzmfdnf-candidate-v9km2q--managed/aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2f2k8zzmfdnf-candidate-v9km2q--managed/providers/Microsoft.Network/dnsZones/aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud/A/*.e2e-dns-swrng7
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-swrng7' does not exist in resource group 'rg-candidate-4-23-2f2k8zzmfdnf-candidate-v9km2q--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-2f2k8zzmfdnf-candidate-v9km2q--managed/providers/Microsoft.Network/dnsZones/aro.r6b2z1c7z7d0i9t.kvy2.j8974848.hcp.osadev.cloud/A/*.e2e-dns-swrng7
            correlation ID: f5ba2020-eed4-41d4-be4e-2d7ff36da168
            client request ID: unavailable
occurred
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded
11:14:07 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-krfsmn"
Unexpected error:
    <*fmt.wrapError | 0x34ab4bc30760>: 
    DNS for route host agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x34ab4c426760>{
            msg: "DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-krfsmn"
Unexpected error:
    <*fmt.wrapError | 0x34ab4bc30760>: 
    DNS for route host agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x34ab4c426760>{
            msg: "DNS for agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-hlljf.apps.aro.c5q3m7v0m2r2b6f.va75.j8974848.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
VerifyCloudNetworkConfigEgressIP(<ip>) failed: failed to label node <node> egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "<node>": nodes "<node>" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not
11:14:07 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.1 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x4d58d"
Unexpected error:
    <*errors.joinError | 0x2a036e0a6030>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc": nodes "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x2a036cc0d0a0>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x2a036cc0d080>{
                    msg: "failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x2a036cc0d060>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x2a036cc3c780>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-5-1-x4d58d"
Unexpected error:
    <*errors.joinError | 0x2a036e0a6030>: 
    VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc": nodes "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized
    {
        errs: [
            <*fmt.wrapError | 0x2a036cc0d0a0>{
                msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.7) failed: failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                err: <*fmt.wrapError | 0x2a036cc0d080>{
                    msg: "failed to label node t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc egress-assignable: failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                    err: <*fmt.wrapError | 0x2a036cc0d060>{
                        msg: "failed to apply k8s.ovn.org/egress-assignable label to node \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\": nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                        err: <*errors.StatusError | 0x2a036cc3c780>{
                            ErrStatus: {
                                TypeMeta: {Kind: "", APIVersion: ""},
                                ListMeta: {
                                    SelfLink: "",
                                    ResourceVersion: "",
                                    Continue: "",
                                    RemainingItemCount: nil,
                                    ShardInfo: nil,
                                },
                                Status: "Failure",
                                Message: "nodes \"t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc\" is forbidden: autoscaling.openshift.io/ManagedNode infra config cache not synchronized",
                                Reason: "Forbidden",
                                Details: {
                                    Name: "t5n3k9o3y9v8w3y-np-x4d58d-tdn4x-htxvc",
                                    Group: "",
                                    Kind: "nodes",
                                    UID: "",
                                    Causes: nil,
                                    RetryAfterSeconds: 0,
                                },
                                Code: 403,
                            },
                        },
                    },
                },
            },
        ],
    }
occurred
2h 21m 16s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head e7c23ba
 centralus e2e failed ⚠#7704 (open) 1
e2e (1)
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: no customer resource ID -> live ReadDesire HostedCluster ->
11:13:52 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8jc7r25jfq6n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xd26a6364d80>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8jc7r25jfq6n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8jc7r25jfq6n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0xd26a6364d60>{
            observation: <*errors.errorString | 0xd26a57e61b0>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0xd26a6364d40>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j6172544-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnpie08b6e3m8ktaflq60p9p59bkmrn-q5i5v4f4b9e4k5j\"; let start = datetime(2026-10-28T12:14:00.5191321Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0xd26a5820dc0>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0xd26a6364d20>{
                        msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j6172544-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnpie08b6e3m8ktaflq60p9p59bkmrn-q5i5v4f4b9e4k5j\"; let start = datetime(2026-10-28T12:14:00.5191321Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0xd26a719e570>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8jc7r25jfq6n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xd26a6364d80>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8jc7r25jfq6n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8jc7r25jfq6n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.006s: no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
        err: <*verifiers.pollObservationError | 0xd26a6364d60>{
            observation: <*errors.errorString | 0xd26a57e61b0>{
                s: "no customer resource ID -> live ReadDesire HostedCluster -> fleet management cluster mapping in Kusto",
            },
            interruption: <*fmt.wrapError | 0xd26a6364d40>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j6172544-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnpie08b6e3m8ktaflq60p9p59bkmrn-q5i5v4f4b9e4k5j\"; let start = datetime(2026-10-28T12:14:00.5191321Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0xd26a5820dc0>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0xd26a6364d20>{
                        msg: "With query: let env = \"dev\"; let rgn = \"centralus\"; let cl = \"ci01-j6172544-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnpie08b6e3m8ktaflq60p9p59bkmrn-q5i5v4f4b9e4k5j\"; let start = datetime(2026-10-28T12:14:00.5191321Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0xd26a719e570>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
3m 35s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 4ec1091 · merge 4ec1091
 n/a unknown failed #7782 (merged) 1 non-artifact-backed failure (no junit artifacts)
2h 13m 28s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head a3c518a · merge a3c518a
 centralus e2e failed #7783 (merged) 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available. Refer ... message truncated; provider Microsoft.RedHatOpenShift
10:45:24 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0x315531e198c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-6tc6qf9c7brh" to finish creating: GET https://rp.j6407296.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ebc64ca-e27a-4569-a3bf-84d10df15019
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ebc64ca-e27a-4569-a3bf-84d10df15019",
      "name": "4ebc64ca-e27a-4569-a3bf-84d10df15019",
      "status": "Failed",
      "startTime": "2026-10-28T11:51:14.858832489Z",
      "endTime": "2026-10-28T12:10:21.145603579Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u9f3h2h7j3o5r2t-np-1-frcss-9d6tp: Failed: interfaces failed to create or update. err: failed to get existing resource pullsecret-test-6tc6qf9c7brh--managed/u9f3h2h7j3o5r2t-np-1-frcss-9d6tp-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-6tc6qf9c7brh--managed/providers/Microsoft.Network/networkInterfaces/u9f3h2h7j3o5r2t-np-1-frcss-9d6tp-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'f8be0ede-39df-41cf-aed7-7f3626f23a5a' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6407296.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ebc64ca-e27a-4569-a3bf-84d10df15019
            correlation ID: 80c82865-2a09-4afd-85b7-df3733e6ccd3
            client request ID: 14042924-6e78-44be-a159-f0ae94071a00
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0x315531e198c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-6tc6qf9c7brh" to finish creating: GET https://rp.j6407296.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ebc64ca-e27a-4569-a3bf-84d10df15019
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ebc64ca-e27a-4569-a3bf-84d10df15019",
      "name": "4ebc64ca-e27a-4569-a3bf-84d10df15019",
      "status": "Failed",
      "startTime": "2026-10-28T11:51:14.858832489Z",
      "endTime": "2026-10-28T12:10:21.145603579Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine u9f3h2h7j3o5r2t-np-1-frcss-9d6tp: Failed: interfaces failed to create or update. err: failed to get existing resource pullsecret-test-6tc6qf9c7brh--managed/u9f3h2h7j3o5r2t-np-1-frcss-9d6tp-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-6tc6qf9c7brh--managed/providers/Microsoft.Network/networkInterfaces/u9f3h2h7j3o5r2t-np-1-frcss-9d6tp-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'f8be0ede-39df-41cf-aed7-7f3626f23a5a' exceeded. Please try again after '10' seconds after additional tokens are available. Refer ... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6407296.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4ebc64ca-e27a-4569-a3bf-84d10df15019
            correlation ID: 80c82865-2a09-4afd-85b7-df3733e6ccd3
            client request ID: 14042924-6e78-44be-a159-f0ae94071a00
occurred
2h 24m 15s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 6fe0578 · merge 6fe0578
 westus3 n/a passed #7795 (merged) 0 n/a
2h 22m 5s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 157b5da
 westus3 alert, e2e failed ⚠#7662 (open) 2
e2e (1)
missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7169, acm-agent=0
09:35:25 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.001s.
Expected success, but got an error:
    <*errors.errorString | 0x2de4f1b0a5d0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7169, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7169, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.001s.
Expected success, but got an error:
    <*errors.errorString | 0x2de4f1b0a5d0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7169, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7169, acm-agent=0",
    }
alert (1)
alert [svc] KubePodCrashLooping fired
09:35:25 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubePodCrashLooping does not fire · suite aro-hcp-tests
Show raw failure
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-28T10:42:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1445504-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.26:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1445504/providers/microsoft.monitor/accounts/services-j1445504", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-5dmcb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6520e185-94fe-45c4-99a9-813a1698d907"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-5dmcb (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-28T10:43:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1445504-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.97:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1445504/providers/microsoft.monitor/accounts/services-j1445504", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-ts4n7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0a28f799-e0bd-4698-b612-8c6756c4b517"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-ts4n7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-28T10:43:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1445504-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.97:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1445504/providers/microsoft.monitor/accounts/services-j1445504", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-ts4n7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0a28f799-e0bd-4698-b612-8c6756c4b517"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-ts4n7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-28T10:47:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1445504-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.26:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1445504/providers/microsoft.monitor/accounts/services-j1445504", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-5dmcb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6520e185-94fe-45c4-99a9-813a1698d907"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-5dmcb (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
2h 13m 54s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head de22b69 · merge de22b69
 westus3 e2e failed ★#6718 (merged) 1
e2e (1)
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j9298304-mgmt-2/ocm-arohcpci01-<id>: expected
09:34:31 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wbzzkmqrv5j9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x6750650e460>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wbzzkmqrv5j9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j9298304-mgmt-2/ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wbzzkmqrv5j9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j9298304-mgmt-2/ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*verifiers.pollObservationError | 0x6750650e400>{
            observation: <*errors.errorString | 0x6750816aa40>{
                s: "dev/westus3/ci01-j9298304-mgmt-2/ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
            },
            interruption: <*fmt.wrapError | 0x6750650e3e0>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9298304-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h\"; let start = datetime(2026-10-28T10:25:27.2654774Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x67506232820>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x6750650e3c0>{
                        msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9298304-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h\"; let start = datetime(2026-10-28T10:25:27.2654774Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x67507892480>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wbzzkmqrv5j9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x6750650e460>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wbzzkmqrv5j9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j9298304-mgmt-2/ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-wbzzkmqrv5j9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: dev/westus3/ci01-j9298304-mgmt-2/ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
        err: <*verifiers.pollObservationError | 0x6750650e400>{
            observation: <*errors.errorString | 0x6750816aa40>{
                s: "dev/westus3/ci01-j9298304-mgmt-2/ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h: expected all current routers >= desired replicas > 0; desired=3 current=2 rows=2",
            },
            interruption: <*fmt.wrapError | 0x6750650e3e0>{
                msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9298304-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h\"; let start = datetime(2026-10-28T10:25:27.2654774Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                err: <*errors.Error | 0x67506232820>{
                    Op: 1,
                    Kind: 7,
                    Err: <*fmt.wrapError | 0x6750650e3c0>{
                        msg: "With query: let env = \"dev\"; let rgn = \"westus3\"; let cl = \"ci01-j9298304-mgmt-2\"; let ns = \"ocm-arohcpci01-2tnnvmiqd0ho3pieph8vtukoridbmhhh-t4k1z6w0y4s5q4h\"; let start = datetime(2026-10-28T10:25:27.2654774Z);\nlet scope = strcat(env, \"/\", rgn, \"/\", cl, \"/\", ns);\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and environment == env and region == rgn and cluster == cl and namespace == ns\n| where objectKind in (\"Pod\", \"Deployment\")\n| summarize arg_max(timestamp, *) by objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project UID = uid, Node = tostring(object.spec.nodeName);\nlet desired = toscalar(inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project tolong(object.spec.replicas));\nlet current = toscalar(pods | count);\nlet summaries = containerLogs\n| where timestamp >= start and environment == env and region == rgn and cluster == cl\n| where container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| where tostring(log.pod_namespace) == ns\n| extend UID = tostring(log.pod_uid)\n| summarize arg_max(timestamp, *) by UID\n| project UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\npods\n| join kind=leftouter summaries on UID\n| extend Scope = scope, Desired = coalesce(desired, tolong(0)), Current = coalesce(current, tolong(0))\n| project Scope, Desired, Current, UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                        err: <*url.Error | 0x67507892480>{
                            Op: "Post",
                            URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                            Err: <context.deadlineExceededError>{},
                        },
                    },
                    restErrMsg: nil,
                    decoded: nil,
                    permanent: false,
                    inner: nil,
                },
            },
        },
    }
occurred
2h 14m 34s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 3b33847 · merge 3b33847
 westus3 n/a passed #7775 (merged) 0 n/a
2h 11m 35s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 558c3f4 · merge 558c3f4
 canadacentral n/a passed #7797 (merged) 0 n/a
2h 37m 42s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head a2f066d · merge a2f066d
 westus3 n/a passed #7706 (merged) 0 n/a
2h 15m 12s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head e7c23ba
 westus3 e2e failed ⚠#7704 (open) 1
e2e (1)
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2262528-mgmt-1/ocm-arohcpci01-<id>: expected all
08:46:09 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8qcj8gl78j5j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2b13ff3f4aa0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8qcj8gl78j5j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2262528-mgmt-1/ocm-arohcpci01-2tnnatgpnn1ff4v4ufusun418oclrbge-r6d7p9b2f9o5i3m: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8qcj8gl78j5j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2262528-mgmt-1/ocm-arohcpci01-2tnnatgpnn1ff4v4ufusun418oclrbge-r6d7p9b2f9o5i3m: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
        err: <*errors.errorString | 0x2b13ffa1a410>{
            s: "dev/westus3/ci01-j2262528-mgmt-1/ocm-arohcpci01-2tnnatgpnn1ff4v4ufusun418oclrbge-r6d7p9b2f9o5i3m: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8qcj8gl78j5j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2b13ff3f4aa0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8qcj8gl78j5j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2262528-mgmt-1/ocm-arohcpci01-2tnnatgpnn1ff4v4ufusun418oclrbge-r6d7p9b2f9o5i3m: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-8qcj8gl78j5j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: dev/westus3/ci01-j2262528-mgmt-1/ocm-arohcpci01-2tnnatgpnn1ff4v4ufusun418oclrbge-r6d7p9b2f9o5i3m: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
        err: <*errors.errorString | 0x2b13ffa1a410>{
            s: "dev/westus3/ci01-j2262528-mgmt-1/ocm-arohcpci01-2tnnatgpnn1ff4v4ufusun418oclrbge-r6d7p9b2f9o5i3m: expected all current routers >= desired replicas > 0; desired=3 current=1 rows=1",
        },
    }
occurred
2h 15m 23s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head be7b63f
 westus3 n/a passed #7798 (open) 0 n/a
2h 14m 41s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 3903554 · merge 3903554
 canadacentral e2e failed ★#7786 (merged) 1
e2e (1)
ERROR CODE: NotFound; detail message The resource record '*.e2e-dns-lpggdd' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network
07:47:20 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-21-tdqwb6"
Unexpected error:
    <*errors.joinError | 0x122a895e0150>: 
    VerifyIngressDNSRecord(aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud) timed out after 10m0.001s: DNS A record "*.e2e-dns-lpggdd" not yet present in zone rg-candidate-4-21-qhznhv86fzbg-candidate-tdqwb6--managed/aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qhznhv86fzbg-candidate-tdqwb6--managed/providers/Microsoft.Network/dnsZones/aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud/A/*.e2e-dns-lpggdd
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-lpggdd' does not exist in resource group 'rg-candidate-4-21-qhznhv86fzbg-candidate-tdqwb6--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qhznhv86fzbg-candidate-tdqwb6--managed/providers/Microsoft.Network/dnsZones/aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud/A/*.e2e-dns-lpggdd
            correlation ID: fc55c026-0ddc-4f29-bb3d-f393fa0f3cd6
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:289]: failed to verify cluster operators on cluster "cluster-candidate-4-21-tdqwb6"
Unexpected error:
    <*errors.joinError | 0x122a895e0150>: 
    VerifyIngressDNSRecord(aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud) timed out after 10m0.001s: DNS A record "*.e2e-dns-lpggdd" not yet present in zone rg-candidate-4-21-qhznhv86fzbg-candidate-tdqwb6--managed/aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qhznhv86fzbg-candidate-tdqwb6--managed/providers/Microsoft.Network/dnsZones/aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud/A/*.e2e-dns-lpggdd
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: NotFound
    --------------------------------------------------------------------------------
    {
      "code": "NotFound",
      "message": "The resource record '*.e2e-dns-lpggdd' does not exist in resource group 'rg-candidate-4-21-qhznhv86fzbg-candidate-tdqwb6--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'."
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-qhznhv86fzbg-candidate-tdqwb6--managed/providers/Microsoft.Network/dnsZones/aro.r1n2g0k6g2s3t3a.soq5.j8963712.hcp.osadev.cloud/A/*.e2e-dns-lpggdd
            correlation ID: fc55c026-0ddc-4f29-bb3d-f393fa0f3cd6
            client request ID: unavailable
occurred
2h 4m 12s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 558c3f4 · merge 558c3f4
 centralus e2e failed #7797 (merged) 1
e2e (1)
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded
07:14:45 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-5pj7h6"
Unexpected error:
    <*fmt.wrapError | 0x2590ea2143e0>: 
    DNS for route host agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x2590e99482e0>{
            msg: "DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:116]: failed to verify simple web app runs on cluster "cluster-hs-pre-5pj7h6"
Unexpected error:
    <*fmt.wrapError | 0x2590ea2143e0>: 
    DNS for route host agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x2590e99482e0>{
            msg: "DNS for agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-dsbx7.apps.aro.m8c7u6n4b6i4x7c.8inl.j9099392.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
2h 5m 49s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 157b5da
 westus3 alert, e2e failed ⚠#7662 (open) 6
e2e (2)
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
07:07:29 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--95r4dzfzp4rt/cluster-hs-pre-5kmssk should provision
Unexpected error:
    <*fmt.wrapError | 0xc9a4e052980>: 
    failed to create HCP cluster cluster-hs-pre-5kmssk: failed waiting for cluster="cluster-hs-pre-5kmssk" in resourcegroup="rg-hs-pre--95r4dzfzp4rt" to finish creating: GET https://rp.j3004160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cb74587-080f-411c-9e53-c65b0814acf7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cb74587-080f-411c-9e53-c65b0814acf7",
      "name": "8cb74587-080f-411c-9e53-c65b0814acf7",
      "status": "Failed",
      "startTime": "2026-10-28T07:54:15.324506005Z",
      "endTime": "2026-10-28T08:23:21.521188973Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3004160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cb74587-080f-411c-9e53-c65b0814acf7
            correlation ID: bc1ad563-f995-498c-a620-7b1f1cac6242
            client request ID: b82578c2-e97e-4b33-9a8e-529b76f26a9b
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--95r4dzfzp4rt/cluster-hs-pre-5kmssk should provision
Unexpected error:
    <*fmt.wrapError | 0xc9a4e052980>: 
    failed to create HCP cluster cluster-hs-pre-5kmssk: failed waiting for cluster="cluster-hs-pre-5kmssk" in resourcegroup="rg-hs-pre--95r4dzfzp4rt" to finish creating: GET https://rp.j3004160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cb74587-080f-411c-9e53-c65b0814acf7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cb74587-080f-411c-9e53-c65b0814acf7",
      "name": "8cb74587-080f-411c-9e53-c65b0814acf7",
      "status": "Failed",
      "startTime": "2026-10-28T07:54:15.324506005Z",
      "endTime": "2026-10-28T08:23:21.521188973Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3004160.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8cb74587-080f-411c-9e53-c65b0814acf7
            correlation ID: bc1ad563-f995-498c-a620-7b1f1cac6242
            client request ID: b82578c2-e97e-4b33-9a8e-529b76f26a9b
occurred
missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7034, acm-agent=0
07:07:29 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.000s.
Expected success, but got an error:
    <*errors.errorString | 0x20bd1e04e7a0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7034, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7034, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.000s.
Expected success, but got an error:
    <*errors.errorString | 0x20bd1e04e7a0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7034, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=7034, acm-agent=0",
    }
alert (4)
alert [hcp] KubeStatefulSetReplicasMismatch fired
07:07:29 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T08:17:22Z
  Ended:       2026-10-28T08:40:22Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/hcps-j3004160", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s", pod="arohcp-monitor-kube-state-metrics-68ff96c687-7dfbl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-28T08:17:22Z
  Ended:       2026-10-28T08:40:22Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/hcps-j3004160", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s", pod="arohcp-monitor-kube-state-metrics-68ff96c687-7dfbl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert [svc] KubePodCrashLooping fired
07:07:29 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubePodCrashLooping does not fire · suite aro-hcp-tests
Show raw failure
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-28T08:13:15Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/services-j3004160", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-krfd7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="2298911c-0832-493f-a9ef-f65728646e11"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-krfd7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-28T08:13:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/services-j3004160", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-krfd7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="2298911c-0832-493f-a9ef-f65728646e11"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-krfd7 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-28T08:15:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.208:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/services-j3004160", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-fgmvz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e461328c-3664-4b8e-907f-f8f42456ba46"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-fgmvz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-28T08:15:13Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.208:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/services-j3004160", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-fgmvz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e461328c-3664-4b8e-907f-f8f42456ba46"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-fgmvz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
alert [hcp] KubePodCrashLooping fired
07:07:29 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T08:20:17Z
  Ended:       2026-10-28T08:44:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/hcps-j3004160", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a66d88b1-9c74-40e9-87a6-eb699288069f"
  Description: Pod ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-28T08:20:17Z
  Ended:       2026-10-28T08:44:17Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/hcps-j3004160", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a66d88b1-9c74-40e9-87a6-eb699288069f"
  Description: Pod ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
alert [hcp] KubeDeploymentReplicasMismatch fired
07:07:29 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-28T08:32:18Z
  Ended:       2026-10-28T08:41:20Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/hcps-j3004160", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s", pod="arohcp-monitor-kube-state-metrics-68ff96c687-7dfbl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-28T08:32:18Z
  Ended:       2026-10-28T08:42:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3004160-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.217:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3004160/providers/microsoft.monitor/accounts/hcps-j3004160", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s", pod="arohcp-monitor-kube-state-metrics-68ff96c687-7dfbl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tnloambpnud3aq6p4sg9mdteg909a3n-z3x1x9z7m5g8p6s/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
2h 17m 14s pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head f4e4b70 · merge 4ec1091
 canadacentral n/a passed #7782 (merged) 0 n/a
2h 14m 20s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 8ae9884 · merge 8ae9884
 westus3 n/a passed #7796 (merged) 0 n/a
2h 26m 32s pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 655931e · merge 6fe0578
 centralus e2e failed #7795 (merged) 1
e2e (1)
ERROR CODE: DeadlineExceeded; detail code SubscriptionRequestsThrottled; detail message Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '<uuid>' exceeded. Please try again after '<seconds>' seconds after additional tokens are available.; provider Microsoft.RedHatOpenShift
05:51:58 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x1ecbca3c5be0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-p9l8lv6w4lzt" to finish creating: GET https://rp.j5142016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d",
      "name": "0d4205c1-5db7-479b-92dc-43f88aa7a88d",
      "status": "Failed",
      "startTime": "2026-10-28T06:54:59.774467587Z",
      "endTime": "2026-10-28T07:14:00.6647054Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-mxkn8-rr6bs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-p9l8lv6w4lzt--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-mxkn8-rr6bs to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20260901-np-1-mxkn8-c8z72: Failed: interfaces failed to create or update. err: failed to get existing resource v20260901preview-p9l8lv6w4lzt--managed/v20260901-np-1-mxkn8-c8z72-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-p9l8lv6w4lzt--managed/providers/Microsoft.Network/networkInterfaces/v20260901-np-1-mxkn8-c8z72-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'db27175c-5bd0-48b4-929a-41de9a53ffbf' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.m... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5142016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d
            correlation ID: 5492e0f1-946f-49bb-8474-ef7f078e1f6f
            client request ID: 12bebe6d-b861-48ec-942f-b913321fe64f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:102]: failed to create node pool "np-1" for v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x1ecbca3c5be0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20260901" in resourcegroup="v20260901preview-p9l8lv6w4lzt" to finish creating: GET https://rp.j5142016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d",
      "name": "0d4205c1-5db7-479b-92dc-43f88aa7a88d",
      "status": "Failed",
      "startTime": "2026-10-28T06:54:59.774467587Z",
      "endTime": "2026-10-28T07:14:00.6647054Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20260901-np-1-mxkn8-rr6bs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-p9l8lv6w4lzt--managed/providers/Microsoft.Compute/virtualMachines/v20260901-np-1-mxkn8-rr6bs to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20260901-np-1-mxkn8-c8z72: Failed: interfaces failed to create or update. err: failed to get existing resource v20260901preview-p9l8lv6w4lzt--managed/v20260901-np-1-mxkn8-c8z72-nic (service: interfaces): GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-p9l8lv6w4lzt--managed/providers/Microsoft.Network/networkInterfaces/v20260901-np-1-mxkn8-c8z72-nic\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor 'db27175c-5bd0-48b4-929a-41de9a53ffbf' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.m... message truncated\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5142016.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0d4205c1-5db7-479b-92dc-43f88aa7a88d
            correlation ID: 5492e0f1-946f-49bb-8474-ef7f078e1f6f
            client request ID: 12bebe6d-b861-48ec-942f-b913321fe64f
occurred
2h 16m 58s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head fbfc1ea
 canadacentral alert, e2e failed ⚠#7662 (open) 6
e2e (5)
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
03:31:21 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x19a135b620a8>: 
    resource group "feature-aggregation-c29rndpv7j5c": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-c29rndpv7j5c" to finish deleting: GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/541f32be-fa69-4e38-80a2-84651b9746cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/541f32be-fa69-4e38-80a2-84651b9746cc",
      "name": "541f32be-fa69-4e38-80a2-84651b9746cc",
      "status": "Failed",
      "startTime": "2026-10-28T04:58:24.120121666Z",
      "endTime": "2026-10-28T05:22:31.687389775Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/541f32be-fa69-4e38-80a2-84651b9746cc
            correlation ID: efcef01e-78c0-4c18-a8a7-6bed6c2a5d41
            client request ID: 23dd3444-0c3c-438b-a21b-7fd27b67a689
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x19a135b620a8>: 
    resource group "feature-aggregation-c29rndpv7j5c": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-c29rndpv7j5c" to finish deleting: GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/541f32be-fa69-4e38-80a2-84651b9746cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/541f32be-fa69-4e38-80a2-84651b9746cc",
      "name": "541f32be-fa69-4e38-80a2-84651b9746cc",
      "status": "Failed",
      "startTime": "2026-10-28T04:58:24.120121666Z",
      "endTime": "2026-10-28T05:22:31.687389775Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/541f32be-fa69-4e38-80a2-84651b9746cc
            correlation ID: efcef01e-78c0-4c18-a8a7-6bed6c2a5d41
            client request ID: 23dd3444-0c3c-438b-a21b-7fd27b67a689
occurred
missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14276, acm-agent=0
03:31:21 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.005s.
Expected success, but got an error:
    <*errors.errorString | 0x867032a97d0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14276, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14276, acm-agent=0",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.005s.
Expected success, but got an error:
    <*errors.errorString | 0x867032a97d0>: 
    missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14276, acm-agent=0
    {
        s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14276, acm-agent=0",
    }
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift
03:31:21 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0xf5ad11741b0>: 
    resource group "rg-candidate-5-0-lf2cjh6fwp8g": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-28vlzn" in resourcegroup="rg-candidate-5-0-lf2cjh6fwp8g" to finish deleting: GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee641f30-7b7c-4096-8a0a-8e212ff6df0d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee641f30-7b7c-4096-8a0a-8e212ff6df0d",
      "name": "ee641f30-7b7c-4096-8a0a-8e212ff6df0d",
      "status": "Failed",
      "startTime": "2026-10-28T04:57:47.524111456Z",
      "endTime": "2026-10-28T05:21:51.687390361Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee641f30-7b7c-4096-8a0a-8e212ff6df0d
            correlation ID: 57653937-379d-4e91-8cdc-72bb96f14f99
            client request ID: da17b50d-815e-4f7b-b6a1-511dd538a3b6
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0xf5ad11741b0>: 
    resource group "rg-candidate-5-0-lf2cjh6fwp8g": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-28vlzn" in resourcegroup="rg-candidate-5-0-lf2cjh6fwp8g" to finish deleting: GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee641f30-7b7c-4096-8a0a-8e212ff6df0d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee641f30-7b7c-4096-8a0a-8e212ff6df0d",
      "name": "ee641f30-7b7c-4096-8a0a-8e212ff6df0d",
      "status": "Failed",
      "startTime": "2026-10-28T04:57:47.524111456Z",
      "endTime": "2026-10-28T05:21:51.687390361Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ee641f30-7b7c-4096-8a0a-8e212ff6df0d
            correlation ID: 57653937-379d-4e91-8cdc-72bb96f14f99
            client request ID: da17b50d-815e-4f7b-b6a1-511dd538a3b6
occurred
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift
03:31:21 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x19fdcb2b4318>: 
    resource group "rg-candidate-4-22-b4lvcfvhwx52": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-jsq2fr" in resourcegroup="rg-candidate-4-22-b4lvcfvhwx52" to finish deleting: GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6f392de4-c202-4b15-9624-974c650cb629
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6f392de4-c202-4b15-9624-974c650cb629",
      "name": "6f392de4-c202-4b15-9624-974c650cb629",
      "status": "Failed",
      "startTime": "2026-10-28T04:51:16.927309848Z",
      "endTime": "2026-10-28T05:15:21.671423449Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6f392de4-c202-4b15-9624-974c650cb629
            correlation ID: 6a7c51a3-41c5-4f0f-a544-ee94e68fe93c
            client request ID: cb15b9e0-1085-4828-b59c-20b42a78c8ff
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x19fdcb2b4318>: 
    resource group "rg-candidate-4-22-b4lvcfvhwx52": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-jsq2fr" in resourcegroup="rg-candidate-4-22-b4lvcfvhwx52" to finish deleting: GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6f392de4-c202-4b15-9624-974c650cb629
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6f392de4-c202-4b15-9624-974c650cb629",
      "name": "6f392de4-c202-4b15-9624-974c650cb629",
      "status": "Failed",
      "startTime": "2026-10-28T04:51:16.927309848Z",
      "endTime": "2026-10-28T05:15:21.671423449Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6f392de4-c202-4b15-9624-974c650cb629
            correlation ID: 6a7c51a3-41c5-4f0f-a544-ee94e68fe93c
            client request ID: cb15b9e0-1085-4828-b59c-20b42a78c8ff
occurred
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift
03:31:21 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x2fb9767520c0>: 
    resource group "rg-candidate-4-23-8gsj9lgrgthl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-x7cmb2" in resourcegroup="rg-candidate-4-23-8gsj9lgrgthl" to finish deleting: GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c5e082c0-738b-4be7-9ea3-e60e834b86c8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c5e082c0-738b-4be7-9ea3-e60e834b86c8",
      "name": "c5e082c0-738b-4be7-9ea3-e60e834b86c8",
      "status": "Failed",
      "startTime": "2026-10-28T04:49:53.842986291Z",
      "endTime": "2026-10-28T05:14:02.592266097Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c5e082c0-738b-4be7-9ea3-e60e834b86c8
            correlation ID: 4e7a26de-7852-4792-8175-683cd23ec9a8
            client request ID: a8483675-393f-439a-880f-cb8cebaae1a8
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:321]: Unexpected error:
    <*errors.joinError | 0x2fb9767520c0>: 
    resource group "rg-candidate-4-23-8gsj9lgrgthl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-x7cmb2" in resourcegroup="rg-candidate-4-23-8gsj9lgrgthl" to finish deleting: GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c5e082c0-738b-4be7-9ea3-e60e834b86c8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c5e082c0-738b-4be7-9ea3-e60e834b86c8",
      "name": "c5e082c0-738b-4be7-9ea3-e60e834b86c8",
      "status": "Failed",
      "startTime": "2026-10-28T04:49:53.842986291Z",
      "endTime": "2026-10-28T05:14:02.592266097Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6816256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c5e082c0-738b-4be7-9ea3-e60e834b86c8
            correlation ID: 4e7a26de-7852-4792-8175-683cd23ec9a8
            client request ID: a8483675-393f-439a-880f-cb8cebaae1a8
occurred
alert (1)
alert [svc] KubePodCrashLooping fired
03:31:21 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubePodCrashLooping does not fire · suite aro-hcp-tests
Show raw failure
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-28T04:40:25Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6816256-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.174:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6816256/providers/microsoft.monitor/accounts/services-j6816256", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-d2g24", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ff44de8f-6ae7-4cd3-9309-4206336bb4a3"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-d2g24 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-28T04:42:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6816256-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.240:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6816256/providers/microsoft.monitor/accounts/services-j6816256", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-mw5bs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="911981bf-9bec-4f85-9bdd-db8c0cfd1db3"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-mw5bs (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-28T04:43:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6816256-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.174:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6816256/providers/microsoft.monitor/accounts/services-j6816256", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-d2g24", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="ff44de8f-6ae7-4cd3-9309-4206336bb4a3"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-d2g24 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-28T04:45:26Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j6816256-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.240:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6816256/providers/microsoft.monitor/accounts/services-j6816256", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-7fc5986cf8-mw5bs", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="911981bf-9bec-4f85-9bdd-db8c0cfd1db3"
  Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-7fc5986cf8-mw5bs (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
2h 4m 31s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head e1dbb29 · merge e1dbb29
 centralus n/a passed #7790 (merged) 0 n/a
2h 48m 9s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head fce1cc2
 canadacentral e2e failed #7791 (open) 1
e2e (1)
ERROR CODE: DeploymentFailed; detail code InvalidTemplate; detail message Unable to process template language expressions for resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Authorization/roleAssignments/<uuid>' at line '<line>' and column '<column>'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'; provider Microsoft.Authorization
02:25:45 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:123]: failed to create customer resources for y-stream upgrade cluster "cp-ystream-upgrade-5-0-pfvhlb"
Unexpected error:
    <*fmt.wrapError | 0x5e040752480>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cp-ystream-upgrade-5-0-pfvhlb-4rsh8t" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cp-ystream-upgrade-5-0-pfvhlb-4rsh8t/operationStatuses/08584084473369857031
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-fqpxn7gcfmxn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-hkoaxxduqas2k\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"InvalidTemplate\",\r\n            \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-fqpxn7gcfmxn/providers/Microsoft.Authorization/roleAssignments/1695ac58-5845-59e4-be60-94380c7b782a' at line '1' and column '21698'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n            \"additionalInfo\": [\r\n              {\r\n                \"type\": \"TemplateViolation\",\r\n                \"info\": {\r\n                  \"lineNumber\": 1,\r\n                  \"linePosition\": 21698,\r\n                  \"path\": \"\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cp-ystream-upgrade-5-0-pfvhlb-4rsh8t/operationStatuses/08584084473369857031
            correlation ID: c39b7be0-22bb-49c9-96d8-f53c5e4ad2fd
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:123]: failed to create customer resources for y-stream upgrade cluster "cp-ystream-upgrade-5-0-pfvhlb"
Unexpected error:
    <*fmt.wrapError | 0x5e040752480>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cp-ystream-upgrade-5-0-pfvhlb-4rsh8t" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cp-ystream-upgrade-5-0-pfvhlb-4rsh8t/operationStatuses/08584084473369857031
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-fqpxn7gcfmxn/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-hkoaxxduqas2k\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"InvalidTemplate\",\r\n            \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-fqpxn7gcfmxn/providers/Microsoft.Authorization/roleAssignments/1695ac58-5845-59e4-be60-94380c7b782a' at line '1' and column '21698'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n            \"additionalInfo\": [\r\n              {\r\n                \"type\": \"TemplateViolation\",\r\n                \"info\": {\r\n                  \"lineNumber\": 1,\r\n                  \"linePosition\": 21698,\r\n                  \"path\": \"\"\r\n                }\r\n              }\r\n            ]\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cp-ystream-upgrade-5-0-pfvhlb-4rsh8t/operationStatuses/08584084473369857031
            correlation ID: c39b7be0-22bb-49c9-96d8-f53c5e4ad2fd
            client request ID: unavailable
occurred
54m 3s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head fbfc1ea
 n/a unknown failed #7662 (open) 1 non-artifact-backed failure (no junit artifacts)
2h 12m 37s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 28823e2
 canadacentral n/a passed #7793 (open) 0 n/a
2h 19m 54s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head b67a95a · merge b67a95a
 westus3 n/a passed #7769 (merged) 0 n/a
36m 33s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head fbfc1ea
 centralus alert failed #7662 (open) 3
alert (3)
ERROR CODE: ResourceGroupNotFound; provider Microsoft.Insights
01:41:17 UTC · failed at alert · test [aro-hcp-observability] [infra] alert collection is complete · suite aro-hcp-tests
Show raw failure
alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2704640: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2704640/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j2704640' could not be found."
  }
}
--------------------------------------------------------------------------------
ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement
01:41:17 UTC · failed at alert · test [aro-hcp-observability] [svc] alert collection is complete · suite aro-hcp-tests
Show raw failure
alert collection incomplete
failed to fetch svc alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2704640/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j2704640' could not be found."
  }
}
--------------------------------------------------------------------------------
ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement
01:41:17 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert collection is complete · suite aro-hcp-tests
Show raw failure
alert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2704640/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j2704640' could not be found."
  }
}
--------------------------------------------------------------------------------
2h 14m 53s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 860fe9f
 westus3 alert failed #6081 (open) 1
alert (1)
alert [svc] ClusterCredentialMetricMissing fired
00:14:36 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ClusterCredentialMetricMissing does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-28T01:28:46Z
  Severity:    Sev3
  Labels:      alertname="ClusterCredentialMetricMissing", cluster="ci01-j5915264-svc", severity="warning"
  Description: Credential refresher expiration metric is missing for service cluster ci01-j5915264-svc.