CIHealth
<
Single Day: Oct 14▾
Last 7 Days Weekly: Oct 12 - Oct 18 Sprint: Oct 12 - Oct 25 Single Day: Oct 14
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 66 · Failures matched: 167 · Total runs: 119 · Runs affected: 88

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyIngressControllerScope(External): timeout must be > 0, got 0s
full failure pattern:
VerifyIngressControllerScope(External): timeout must be > 0, got 0s
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:187]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*errors.errorString | 0x26b7f5580860>: 
    VerifyIngressControllerScope(External): timeout must be > 0, got 0s
    {
        s: "VerifyIngressControllerScope(External): timeout must be > 0, got 0s",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:187]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*errors.errorString | 0x26b7f5580860>: 
    VerifyIngressControllerScope(External): timeout must be > 0, got 0s
    {
        s: "VerifyIngressControllerScope(External): timeout must be > 0, got 0s",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:187]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*errors.errorString | 0x5b6d94696d0>: 
    VerifyIngressControllerScope(External): timeout must be > 0, got 0s
    {
        s: "VerifyIngressControllerScope(External): timeout must be > 0, got 0s",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:187]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*errors.errorString | 0x5b6d94696d0>: 
    VerifyIngressControllerScope(External): timeout must be > 0, got 0s
    {
        s: "VerifyIngressControllerScope(External): timeout must be > 0, got 0s",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:187]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*errors.errorString | 0x362088f5cd60>: 
    VerifyIngressControllerScope(External): timeout must be > 0, got 0s
    {
        s: "VerifyIngressControllerScope(External): timeout must be > 0, got 0s",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:187]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*errors.errorString | 0x362088f5cd60>: 
    VerifyIngressControllerScope(External): timeout must be > 0, got 0s
    {
        s: "VerifyIngressControllerScope(External): timeout must be > 0, got 0s",
    }
Contributing tests (4)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 5.11
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74772110340691224694784
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifyIngressControllerScope(External) timed out after 5m0
full failure pattern:
VerifyIngressControllerScope(External) timed out after 5m0
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:199]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*fmt.wrapError | 0x2c6abc8c6320>: 
    VerifyIngressControllerScope(External) timed out after 5m0.002s: failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyIngressControllerScope(External) timed out after 5m0.002s: failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x2c6abc8c62a0>{
            msg: "failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x2c6abc8c6260>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:199]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*fmt.wrapError | 0x2c6abc8c6320>: 
    VerifyIngressControllerScope(External) timed out after 5m0.002s: failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyIngressControllerScope(External) timed out after 5m0.002s: failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x2c6abc8c62a0>{
            msg: "failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x2c6abc8c6260>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:199]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*fmt.wrapError | 0x1ba452ab87c0>: 
    VerifyIngressControllerScope(External) timed out after 5m0.006s: failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyIngressControllerScope(External) timed out after 5m0.006s: failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1ba452ab87a0>{
            msg: "failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1ba452ab8760>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:199]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
    <*fmt.wrapError | 0x1ba452ab87c0>: 
    VerifyIngressControllerScope(External) timed out after 5m0.006s: failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyIngressControllerScope(External) timed out after 5m0.006s: failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x1ba452ab87a0>{
            msg: "failed to get default IngressController: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x1ba452ab8760>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 4.211
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74772110235250528358400
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-<id> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-<id> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-<id> to exist; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2e793540e048>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-4b88pr2h6f62" to finish creating: GET https://rp.j1691264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02548047-b3fd-49b8-8426-7df711bcbfbe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02548047-b3fd-49b8-8426-7df711bcbfbe",
      "name": "02548047-b3fd-49b8-8426-7df711bcbfbe",
      "status": "Failed",
      "startTime": "2026-10-14T10:01:14.415831901Z",
      "endTime": "2026-10-14T10:30:17.083278262Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-794ws-42z9w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4b88pr2h6f62--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-794ws-42z9w to exist\nMachine cilium-cluster-cilium-np-794ws-44qrb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4b88pr2h6f62--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-794ws-44qrb to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-794ws-42z9w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4b88pr2h6f62--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-794ws-42z9w to exist\nMachine cilium-cluster-cilium-np-794ws-44qrb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4b88pr2h6f62--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-794ws-44qrb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1691264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02548047-b3fd-49b8-8426-7df711bcbfbe
            correlation ID: d40a80e8-ffc2-4aca-bd50-98d7806c38e8
            client request ID: 84b1cf39-1ee3-4c6a-80ac-30f66febe153
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:193]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0x2e793540e048>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-4b88pr2h6f62" to finish creating: GET https://rp.j1691264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02548047-b3fd-49b8-8426-7df711bcbfbe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02548047-b3fd-49b8-8426-7df711bcbfbe",
      "name": "02548047-b3fd-49b8-8426-7df711bcbfbe",
      "status": "Failed",
      "startTime": "2026-10-14T10:01:14.415831901Z",
      "endTime": "2026-10-14T10:30:17.083278262Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine cilium-cluster-cilium-np-794ws-42z9w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4b88pr2h6f62--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-794ws-42z9w to exist\nMachine cilium-cluster-cilium-np-794ws-44qrb: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4b88pr2h6f62--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-794ws-44qrb to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine cilium-cluster-cilium-np-794ws-42z9w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4b88pr2h6f62--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-794ws-42z9w to exist\nMachine cilium-cluster-cilium-np-794ws-44qrb: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-4b88pr2h6f62--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-794ws-44qrb to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1691264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/02548047-b3fd-49b8-8426-7df711bcbfbe
            correlation ID: d40a80e8-ffc2-4aca-bd50-98d7806c38e8
            client request ID: 84b1cf39-1ee3-4c6a-80ac-30f66febe153
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75002110291728471691264
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-rpntb-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-rpntb-<id> to exist; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0x346ef5b24080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-xn2t4zqk2qxz" to finish creating: GET https://rp.j6626944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/439dbd7f-6ac7-4b48-8533-a8815df6b20f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/439dbd7f-6ac7-4b48-8533-a8815df6b20f",
      "name": "439dbd7f-6ac7-4b48-8533-a8815df6b20f",
      "status": "Failed",
      "startTime": "2026-10-14T12:55:20.912229682Z",
      "endTime": "2026-10-14T13:24:20.249835656Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine private-ingress-np-1-rpntb-stl8c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-xn2t4zqk2qxz--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-rpntb-stl8c to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-rpntb-stl8c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-xn2t4zqk2qxz--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-rpntb-stl8c to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6626944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/439dbd7f-6ac7-4b48-8533-a8815df6b20f
            correlation ID: eefc14a4-21ec-420e-9b63-e8df580da42a
            client request ID: b9d8fbaf-fe74-4488-8491-9a8ef5ba8fd7
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0x346ef5b24080>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-xn2t4zqk2qxz" to finish creating: GET https://rp.j6626944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/439dbd7f-6ac7-4b48-8533-a8815df6b20f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/439dbd7f-6ac7-4b48-8533-a8815df6b20f",
      "name": "439dbd7f-6ac7-4b48-8533-a8815df6b20f",
      "status": "Failed",
      "startTime": "2026-10-14T12:55:20.912229682Z",
      "endTime": "2026-10-14T13:24:20.249835656Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine private-ingress-np-1-rpntb-stl8c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-xn2t4zqk2qxz--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-rpntb-stl8c to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-rpntb-stl8c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-xn2t4zqk2qxz--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-rpntb-stl8c to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6626944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/439dbd7f-6ac7-4b48-8533-a8815df6b20f
            correlation ID: eefc14a4-21ec-420e-9b63-e8df580da42a
            client request ID: b9d8fbaf-fe74-4488-8491-9a8ef5ba8fd7
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75252110338041116626944
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-fqbnw to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-qvbld to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-fqbnw to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-qvbld to exist; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xcf2616b6280>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-mfxbjq8jns8s" to finish creating: GET https://rp.j9450880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6ef2bbd7-ca39-48e4-90ab-26948b5b65f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6ef2bbd7-ca39-48e4-90ab-26948b5b65f1",
      "name": "6ef2bbd7-ca39-48e4-90ab-26948b5b65f1",
      "status": "Failed",
      "startTime": "2026-10-14T07:49:18.1783953Z",
      "endTime": "2026-10-14T08:18:20.880236175Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-cw6kh-fqbnw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mfxbjq8jns8s--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-fqbnw to exist\nMachine v20261001-np-1-cw6kh-qvbld: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mfxbjq8jns8s--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-qvbld to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-cw6kh-fqbnw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mfxbjq8jns8s--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-fqbnw to exist\nMachine v20261001-np-1-cw6kh-qvbld: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mfxbjq8jns8s--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-qvbld to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9450880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6ef2bbd7-ca39-48e4-90ab-26948b5b65f1
            correlation ID: 39af48df-821f-41b4-9eda-383af9fbc767
            client request ID: d214e485-a699-4f3c-bb9e-2d4e1497eed9
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0xcf2616b6280>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-mfxbjq8jns8s" to finish creating: GET https://rp.j9450880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6ef2bbd7-ca39-48e4-90ab-26948b5b65f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6ef2bbd7-ca39-48e4-90ab-26948b5b65f1",
      "name": "6ef2bbd7-ca39-48e4-90ab-26948b5b65f1",
      "status": "Failed",
      "startTime": "2026-10-14T07:49:18.1783953Z",
      "endTime": "2026-10-14T08:18:20.880236175Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine v20261001-np-1-cw6kh-fqbnw: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mfxbjq8jns8s--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-fqbnw to exist\nMachine v20261001-np-1-cw6kh-qvbld: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mfxbjq8jns8s--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-qvbld to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine v20261001-np-1-cw6kh-fqbnw: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mfxbjq8jns8s--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-fqbnw to exist\nMachine v20261001-np-1-cw6kh-qvbld: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-mfxbjq8jns8s--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-cw6kh-qvbld to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9450880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6ef2bbd7-ca39-48e4-90ab-26948b5b65f1
            correlation ID: 39af48df-821f-41b4-9eda-383af9fbc767
            client request ID: d214e485-a699-4f3c-bb9e-2d4e1497eed9
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #67122110261282979450880
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: SWIFT query row
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6k948d9nsvr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1f63e25da900>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6k948d9nsvr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-15T00:18:45.8286809Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f", ActivityId:"3e4bf140-f035-443b-96b4-bed0d927cd4f", SubActivityId:"3e4bf140-f035-443b-96b4-bed0d927cd4f", ActivityType:"GW.Http.CallContext", ParentActivityId:"3e4bf140-f035-443b-96b4-bed0d927cd4f", ActivityStack:"(Activity stack: CRID=KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f ARID=3e4bf140-f035-443b-96b4-bed0d927cd4f > GW.Http.CallContext/3e4bf140-f035-443b-96b4-bed0d927cd4f)"}, IsPermanent:false})
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6k948d9nsvr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-15T00:18:45.8286809Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f\", ActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", SubActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f ARID=3e4bf140-f035-443b-96b4-bed0d927cd4f > GW.Http.CallContext/3e4bf140-f035-443b-96b4-bed0d927cd4f)\"}, IsPermanent:false})",
        err: <*fmt.wrapError | 0x1f63e28703a0>{
            msg: "SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-15T00:18:45.8286809Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f\", ActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", SubActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f ARID=3e4bf140-f035-443b-96b4-bed0d927cd4f > GW.Http.CallContext/3e4bf140-f035-443b-96b4-bed0d927cd4f)\"}, IsPermanent:false})",
            err: <*v2.OneApiError | 0x1f63e2d64800>{
                ErrorMessage: {
                    Code: "LimitsExceeded",
                    Message: "Request is invalid and cannot be executed.",
                    Description: "Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..",
                    Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                    Context: {
                        Timestamp: "2026-10-15T00:18:45.8286809Z",
                        ServiceAlias: "HCP-DEV-US-2",
                        MachineName: "",
                        ProcessName: "",
                        ProcessId: 0,
                        ThreadId: 0,
                        ClientRequestId: "KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f",
                        ActivityId: "3e4bf140-f035-443b-96b4-bed0d927cd4f",
                        SubActivityId: "3e4bf140-f035-443b-96b4-bed0d927cd4f",
                        ActivityType: "GW.Http.CallContext",
                        ParentActivityId: "3e4bf140-f035-443b-96b4-bed0d927cd4f",
                        ActivityStack: "(Activity stack: CRID=KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f ARID=3e4bf140-f035-443b-96b4-bed0d927cd4f > GW.Http.CallContext/3e4bf140-f035-443b-96b4-bed0d927cd4f)",
                    },
                    IsPermanent: false,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6k948d9nsvr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1f63e25da900>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6k948d9nsvr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-15T00:18:45.8286809Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f", ActivityId:"3e4bf140-f035-443b-96b4-bed0d927cd4f", SubActivityId:"3e4bf140-f035-443b-96b4-bed0d927cd4f", ActivityType:"GW.Http.CallContext", ParentActivityId:"3e4bf140-f035-443b-96b4-bed0d927cd4f", ActivityStack:"(Activity stack: CRID=KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f ARID=3e4bf140-f035-443b-96b4-bed0d927cd4f > GW.Http.CallContext/3e4bf140-f035-443b-96b4-bed0d927cd4f)"}, IsPermanent:false})
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-6k948d9nsvr2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-15T00:18:45.8286809Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f\", ActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", SubActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f ARID=3e4bf140-f035-443b-96b4-bed0d927cd4f > GW.Http.CallContext/3e4bf140-f035-443b-96b4-bed0d927cd4f)\"}, IsPermanent:false})",
        err: <*fmt.wrapError | 0x1f63e28703a0>{
            msg: "SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-15T00:18:45.8286809Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f\", ActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", SubActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"3e4bf140-f035-443b-96b4-bed0d927cd4f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f ARID=3e4bf140-f035-443b-96b4-bed0d927cd4f > GW.Http.CallContext/3e4bf140-f035-443b-96b4-bed0d927cd4f)\"}, IsPermanent:false})",
            err: <*v2.OneApiError | 0x1f63e2d64800>{
                ErrorMessage: {
                    Code: "LimitsExceeded",
                    Message: "Request is invalid and cannot be executed.",
                    Description: "Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..",
                    Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                    Context: {
                        Timestamp: "2026-10-15T00:18:45.8286809Z",
                        ServiceAlias: "HCP-DEV-US-2",
                        MachineName: "",
                        ProcessName: "",
                        ProcessId: 0,
                        ThreadId: 0,
                        ClientRequestId: "KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f",
                        ActivityId: "3e4bf140-f035-443b-96b4-bed0d927cd4f",
                        SubActivityId: "3e4bf140-f035-443b-96b4-bed0d927cd4f",
                        ActivityType: "GW.Http.CallContext",
                        ParentActivityId: "3e4bf140-f035-443b-96b4-bed0d927cd4f",
                        ActivityStack: "(Activity stack: CRID=KGC.execute;e95fc3a4-a2a2-47ad-b9ce-87006b7b438f ARID=3e4bf140-f035-443b-96b4-bed0d927cd4f > GW.Http.CallContext/3e4bf140-f035-443b-96b4-bed0d927cd4f)",
                    },
                    IsPermanent: false,
                },
            },
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74512110505977575903232
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in <url> context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-54nwd6zcc79j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2620bace6480>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-54nwd6zcc79j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-54nwd6zcc79j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x2620bace6460>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-54nwd6zcc79j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x2620bace6480>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-54nwd6zcc79j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-54nwd6zcc79j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x2620bace6460>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75252110338041116626944
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: SWIFT query row
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fxq6rqgzrkpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x21de3cbea400>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fxq6rqgzrkpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-14T11:13:29.6322749Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736", ActivityId:"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd", SubActivityId:"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd", ActivityType:"GW.Http.CallContext", ParentActivityId:"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd", ActivityStack:"(Activity stack: CRID=KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736 ARID=9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd > GW.Http.CallContext/9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd)"}, IsPermanent:false})
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fxq6rqgzrkpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T11:13:29.6322749Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736\", ActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", SubActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736 ARID=9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd > GW.Http.CallContext/9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd)\"}, IsPermanent:false})",
        err: <*fmt.wrapError | 0x21de3ca9e560>{
            msg: "SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T11:13:29.6322749Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736\", ActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", SubActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736 ARID=9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd > GW.Http.CallContext/9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd)\"}, IsPermanent:false})",
            err: <*v2.OneApiError | 0x21de3e2c5500>{
                ErrorMessage: {
                    Code: "LimitsExceeded",
                    Message: "Request is invalid and cannot be executed.",
                    Description: "Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..",
                    Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                    Context: {
                        Timestamp: "2026-10-14T11:13:29.6322749Z",
                        ServiceAlias: "HCP-DEV-US-2",
                        MachineName: "",
                        ProcessName: "",
                        ProcessId: 0,
                        ThreadId: 0,
                        ClientRequestId: "KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736",
                        ActivityId: "9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd",
                        SubActivityId: "9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd",
                        ActivityType: "GW.Http.CallContext",
                        ParentActivityId: "9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd",
                        ActivityStack: "(Activity stack: CRID=KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736 ARID=9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd > GW.Http.CallContext/9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd)",
                    },
                    IsPermanent: false,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fxq6rqgzrkpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x21de3cbea400>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fxq6rqgzrkpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-14T11:13:29.6322749Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736", ActivityId:"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd", SubActivityId:"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd", ActivityType:"GW.Http.CallContext", ParentActivityId:"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd", ActivityStack:"(Activity stack: CRID=KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736 ARID=9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd > GW.Http.CallContext/9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd)"}, IsPermanent:false})
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-fxq6rqgzrkpd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T11:13:29.6322749Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736\", ActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", SubActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736 ARID=9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd > GW.Http.CallContext/9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd)\"}, IsPermanent:false})",
        err: <*fmt.wrapError | 0x21de3ca9e560>{
            msg: "SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T11:13:29.6322749Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736\", ActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", SubActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736 ARID=9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd > GW.Http.CallContext/9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd)\"}, IsPermanent:false})",
            err: <*v2.OneApiError | 0x21de3e2c5500>{
                ErrorMessage: {
                    Code: "LimitsExceeded",
                    Message: "Request is invalid and cannot be executed.",
                    Description: "Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..",
                    Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                    Context: {
                        Timestamp: "2026-10-14T11:13:29.6322749Z",
                        ServiceAlias: "HCP-DEV-US-2",
                        MachineName: "",
                        ProcessName: "",
                        ProcessId: 0,
                        ThreadId: 0,
                        ClientRequestId: "KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736",
                        ActivityId: "9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd",
                        SubActivityId: "9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd",
                        ActivityType: "GW.Http.CallContext",
                        ParentActivityId: "9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd",
                        ActivityStack: "(Activity stack: CRID=KGC.execute;a517dabe-2339-417d-83d4-3b27c7edb736 ARID=9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd > GW.Http.CallContext/9f47d8f2-d8b0-493f-b7da-5e950a8a5dbd)",
                    },
                    IsPermanent: false,
                },
            },
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75222110302904370860032
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in <url> context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-h55twqdpvpzw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xee59296c020>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-h55twqdpvpzw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-h55twqdpvpzw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0xee59296c000>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-h55twqdpvpzw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xee59296c020>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-h55twqdpvpzw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-h55twqdpvpzw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.003s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0xee59296c000>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75232110300965679665152
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.007s: query SWIFT checks in <url> context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mpzr4p97ncr4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x298032100120>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mpzr4p97ncr4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.007s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mpzr4p97ncr4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.007s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x298032100100>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mpzr4p97ncr4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x298032100120>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mpzr4p97ncr4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.007s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-mpzr4p97ncr4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.007s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x298032100100>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74992110452208637054976
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: SWIFT query row
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tfgpxb4klb8k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1dded76126c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tfgpxb4klb8k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-14T11:13:43.5550944Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c", ActivityId:"0eb2baf5-e792-455b-9c86-a0654265ab17", SubActivityId:"0eb2baf5-e792-455b-9c86-a0654265ab17", ActivityType:"GW.Http.CallContext", ParentActivityId:"0eb2baf5-e792-455b-9c86-a0654265ab17", ActivityStack:"(Activity stack: CRID=KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c ARID=0eb2baf5-e792-455b-9c86-a0654265ab17 > GW.Http.CallContext/0eb2baf5-e792-455b-9c86-a0654265ab17)"}, IsPermanent:false})
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tfgpxb4klb8k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T11:13:43.5550944Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c\", ActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", SubActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", ActivityStack:\"(Activity stack: CRID=KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c ARID=0eb2baf5-e792-455b-9c86-a0654265ab17 > GW.Http.CallContext/0eb2baf5-e792-455b-9c86-a0654265ab17)\"}, IsPermanent:false})",
        err: <*fmt.wrapError | 0x1dded76126a0>{
            msg: "SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T11:13:43.5550944Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c\", ActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", SubActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", ActivityStack:\"(Activity stack: CRID=KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c ARID=0eb2baf5-e792-455b-9c86-a0654265ab17 > GW.Http.CallContext/0eb2baf5-e792-455b-9c86-a0654265ab17)\"}, IsPermanent:false})",
            err: <*v2.OneApiError | 0x1dded776cc00>{
                ErrorMessage: {
                    Code: "LimitsExceeded",
                    Message: "Request is invalid and cannot be executed.",
                    Description: "Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..",
                    Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                    Context: {
                        Timestamp: "2026-10-14T11:13:43.5550944Z",
                        ServiceAlias: "HCP-DEV-US-2",
                        MachineName: "",
                        ProcessName: "",
                        ProcessId: 0,
                        ThreadId: 0,
                        ClientRequestId: "KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c",
                        ActivityId: "0eb2baf5-e792-455b-9c86-a0654265ab17",
                        SubActivityId: "0eb2baf5-e792-455b-9c86-a0654265ab17",
                        ActivityType: "GW.Http.CallContext",
                        ParentActivityId: "0eb2baf5-e792-455b-9c86-a0654265ab17",
                        ActivityStack: "(Activity stack: CRID=KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c ARID=0eb2baf5-e792-455b-9c86-a0654265ab17 > GW.Http.CallContext/0eb2baf5-e792-455b-9c86-a0654265ab17)",
                    },
                    IsPermanent: false,
                },
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tfgpxb4klb8k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1dded76126c0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tfgpxb4klb8k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-14T11:13:43.5550944Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c", ActivityId:"0eb2baf5-e792-455b-9c86-a0654265ab17", SubActivityId:"0eb2baf5-e792-455b-9c86-a0654265ab17", ActivityType:"GW.Http.CallContext", ParentActivityId:"0eb2baf5-e792-455b-9c86-a0654265ab17", ActivityStack:"(Activity stack: CRID=KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c ARID=0eb2baf5-e792-455b-9c86-a0654265ab17 > GW.Http.CallContext/0eb2baf5-e792-455b-9c86-a0654265ab17)"}, IsPermanent:false})
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-tfgpxb4klb8k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T11:13:43.5550944Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c\", ActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", SubActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", ActivityStack:\"(Activity stack: CRID=KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c ARID=0eb2baf5-e792-455b-9c86-a0654265ab17 > GW.Http.CallContext/0eb2baf5-e792-455b-9c86-a0654265ab17)\"}, IsPermanent:false})",
        err: <*fmt.wrapError | 0x1dded76126a0>{
            msg: "SWIFT query row: OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T11:13:43.5550944Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c\", ActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", SubActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0eb2baf5-e792-455b-9c86-a0654265ab17\", ActivityStack:\"(Activity stack: CRID=KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c ARID=0eb2baf5-e792-455b-9c86-a0654265ab17 > GW.Http.CallContext/0eb2baf5-e792-455b-9c86-a0654265ab17)\"}, IsPermanent:false})",
            err: <*v2.OneApiError | 0x1dded776cc00>{
                ErrorMessage: {
                    Code: "LimitsExceeded",
                    Message: "Request is invalid and cannot be executed.",
                    Description: "Query execution has exceeded the allowed limits (80DA0001): 'join' operator has exceeded the memory budget (5368709120) during evaluation. Results may be incorrect or incomplete (E_RUNAWAY_QUERY; see https://aka.ms/kustoquerylimits)..",
                    Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                    Context: {
                        Timestamp: "2026-10-14T11:13:43.5550944Z",
                        ServiceAlias: "HCP-DEV-US-2",
                        MachineName: "",
                        ProcessName: "",
                        ProcessId: 0,
                        ThreadId: 0,
                        ClientRequestId: "KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c",
                        ActivityId: "0eb2baf5-e792-455b-9c86-a0654265ab17",
                        SubActivityId: "0eb2baf5-e792-455b-9c86-a0654265ab17",
                        ActivityType: "GW.Http.CallContext",
                        ParentActivityId: "0eb2baf5-e792-455b-9c86-a0654265ab17",
                        ActivityStack: "(Activity stack: CRID=KGC.execute;458198cd-6878-4ea0-8879-a65dd48c7d9c ARID=0eb2baf5-e792-455b-9c86-a0654265ab17 > GW.Http.CallContext/0eb2baf5-e792-455b-9c86-a0654265ab17)",
                    },
                    IsPermanent: false,
                },
            },
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73832110303886584582144
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: query SWIFT checks in <url> context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p2q7sjlczt4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x128374ba040>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p2q7sjlczt4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p2q7sjlczt4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x128374ba020>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p2q7sjlczt4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x128374ba040>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p2q7sjlczt4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-p2q7sjlczt4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.008s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x128374ba020>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74982110305441572458496
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRVerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out af...
full failure pattern:
VerifySwiftRouterChecks /subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: query SWIFT checks in <url> context deadline exceeded
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2t8nlzw5hxlc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x32bc856472e0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2t8nlzw5hxlc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2t8nlzw5hxlc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x32bc85647260>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2t8nlzw5hxlc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x32bc856472e0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2t8nlzw5hxlc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2t8nlzw5hxlc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
        err: <*fmt.wrapError | 0x32bc85647260>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #67462110452257878183936
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRalert [infra] Cosmos DB Normalized RU Consumption Burst High - <cosmos-account> fired
full failure pattern:
alert [infra] Cosmos DB Normalized RU Consumption Burst High - <cosmos-account> fired
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
alert10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 3 · Oct 14: 1none
Full failure examples (1)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-14T03:40:18Z
  Ended:       2026-10-14T03:43:19Z
  Severity:    Sev3
  Description: Cosmos DB normalized RU consumption is above 95% averaged over a 5-minute window, evaluated every minute. This catches short RU-saturation bursts (e.g. mass cluster creation at the start of E2E tests) that exceed provisioned throughput and can lead to 429 throttling before the 10-minute sustained alert fires. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units
Firing 2:
  State:       Resolved
  Started:     2026-10-14T03:52:13Z
  Ended:       2026-10-14T04:06:24Z
  Severity:    Sev3
  Description: Cosmos DB normalized RU consumption is above 95% averaged over a 5-minute window, evaluated every minute. This catches short RU-saturation bursts (e.g. mass cluster creation at the start of E2E tests) that exceed provisioned throughput and can lead to 429 throttling before the 10-minute sustained alert fires. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [infra] alert Cosmos DB Normalized RU Consumption Burst High - arohcpci01-rp-j7369856 does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74952110199338117369856
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRclusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no regis...
full failure pattern:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no registrations with public key sha256 "3vAWYLSpgMpdmMCKnkDG1+GD7QF9y2L9E9AOfsGsY04="
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:364]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0x181b25f73380>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no registrations with public key sha256 "3vAWYLSpgMpdmMCKnkDG1+GD7QF9y2L9E9AOfsGsY04="
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no registrations with public key sha256 \"3vAWYLSpgMpdmMCKnkDG1+GD7QF9y2L9E9AOfsGsY04=\"",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:364]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0x181b25f73380>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no registrations with public key sha256 "3vAWYLSpgMpdmMCKnkDG1+GD7QF9y2L9E9AOfsGsY04="
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: 404 urn:ietf:params:acme:error:malformed: checking for existing account :: no registrations with public key sha256 \"3vAWYLSpgMpdmMCKnkDG1+GD7QF9y2L9E9AOfsGsY04=\"",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75002110389003348873216
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRerror running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>...
full failure pattern:
error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j2981888-mgmt-1","clusterName":"ci01-j298188
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
time=2026-10-14T19:33:50.044Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-14T20:03:49.640Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-14T19:33:50.055321564Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2981888-mgmt-1\",\"clusterName\":\"ci01-j2981888-mgmt-1\",\"region\":\"westus3\"}\n{\"time\":\"2026-10-14T19:33:52.298968498Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2981888-mgmt-1\",\"clusterName\":\"ci01-j2981888-mgmt-1\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-14T19:33:52.677544841Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2981888-mgmt-1\",\"clusterName\":\"ci01-j2981888-mgmt-1\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=1)1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69462110441834562981888
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to gather logs: failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: clustersServiceLogs
full failure pattern:
failed to gather logs: failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: clustersServiceLogs
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 602.098s.
Expected success, but got an error:
    <*fmt.wrapError | 0x3618a9570d00>: 
    failed to gather logs: failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: clustersServiceLogs
    | where resource_id has '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5967pdcp8vw8'
    | distinct cid, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": net/http: TLS handshake timeout
    {
        msg: "failed to gather logs: failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: clustersServiceLogs\n| where resource_id has '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5967pdcp8vw8'\n| distinct cid, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": net/http: TLS handshake timeout",
        err: <*fmt.wrapError | 0x3618a9570cc0>{
            msg: "failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: clustersServiceLogs\n| where resource_id has '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5967pdcp8vw8'\n| distinct cid, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": net/http: TLS handshake timeout",
            err: <*fmt.wrapError | 0x3618a9570c60>{
                msg: "failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: clustersServiceLogs\n| where resource_id has '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5967pdcp8vw8'\n| distinct cid, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": net/http: TLS handshake timeout",
                err: <*fmt.wrapError | 0x3618a9570c40>{
                    msg: "failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: clustersServiceLogs\n| where resource_id has '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5967pdcp8vw8'\n| distinct cid, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": net/http: TLS handshake timeout",
                    err: <*errors.Error | 0x3618a9654280>{
                        Op: 1,
                        Kind: 7,
                        Err: <*fmt.wrapError | 0x3618a9570c00>{
                            msg: "With query: clustersServiceLogs\n| where resource_id has '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5967pdcp8vw8'\n| distinct cid, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": net/http: TLS handshake timeout",
                            err: <*url.Error | 0x3618aa1308a0>{
                                Op: "Post",
                                URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                                Err: <http.tlsHandshakeTimeoutError>{},
                            },
                        },
                        restErrMsg: nil,
                        decoded: nil,
                        permanent: false,
                        inner: nil,
                    },
                },
            },
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 602.098s.
Expected success, but got an error:
    <*fmt.wrapError | 0x3618a9570d00>: 
    failed to gather logs: failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: clustersServiceLogs
    | where resource_id has '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5967pdcp8vw8'
    | distinct cid, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": net/http: TLS handshake timeout
    {
        msg: "failed to gather logs: failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: clustersServiceLogs\n| where resource_id has '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5967pdcp8vw8'\n| distinct cid, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": net/http: TLS handshake timeout",
        err: <*fmt.wrapError | 0x3618a9570cc0>{
            msg: "failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: clustersServiceLogs\n| where resource_id has '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5967pdcp8vw8'\n| distinct cid, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": net/http: TLS handshake timeout",
            err: <*fmt.wrapError | 0x3618a9570c60>{
                msg: "failed to execute query: failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: clustersServiceLogs\n| where resource_id has '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5967pdcp8vw8'\n| distinct cid, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": net/http: TLS handshake timeout",
                err: <*fmt.wrapError | 0x3618a9570c40>{
                    msg: "failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: clustersServiceLogs\n| where resource_id has '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5967pdcp8vw8'\n| distinct cid, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": net/http: TLS handshake timeout",
                    err: <*errors.Error | 0x3618a9654280>{
                        Op: 1,
                        Kind: 7,
                        Err: <*fmt.wrapError | 0x3618a9570c00>{
                            msg: "With query: clustersServiceLogs\n| where resource_id has '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5967pdcp8vw8'\n| distinct cid, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": net/http: TLS handshake timeout",
                            err: <*url.Error | 0x3618aa1308a0>{
                                Op: "Post",
                                URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                                Err: <http.tlsHandshakeTimeoutError>{},
                            },
                        },
                        restErrMsg: nil,
                        decoded: nil,
                        permanent: false,
                        inner: nil,
                    },
                },
            },
        },
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #67122110261282979450880
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRfailed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQ...
full failure pattern:
failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-13T19:48:15
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.007s.
Expected success, but got an error:
    <*fmt.wrapError | 0x356d91091ca0>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-13T19:48:15.7310676Z);
    let _endTime = datetime(2026-10-14T19:48:15.7310678Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-vcw9kpw6mcqc'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-14T19:48:28.4804395Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e", ActivityId:"9181dd6d-5c5a-470f-8a62-6f1f2abde29b", SubActivityId:"9181dd6d-5c5a-470f-8a62-6f1f2abde29b", ActivityType:"GW.Http.CallContext", ParentActivityId:"9181dd6d-5c5a-470f-8a62-6f1f2abde29b", ActivityStack:"(Activity stack: CRID=KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e ARID=9181dd6d-5c5a-470f-8a62-6f1f2abde29b > GW.Http.CallContext/9181dd6d-5c5a-470f-8a62-6f1f2abde29b)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-13T19:48:15.7310676Z);\nlet _endTime = datetime(2026-10-14T19:48:15.7310678Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-vcw9kpw6mcqc'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T19:48:28.4804395Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e\", ActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", SubActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e ARID=9181dd6d-5c5a-470f-8a62-6f1f2abde29b > GW.Http.CallContext/9181dd6d-5c5a-470f-8a62-6f1f2abde29b)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x356d921ef170>{
            errs: [
                <*fmt.wrapError | 0x356d91091c80>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-13T19:48:15.7310676Z);\nlet _endTime = datetime(2026-10-14T19:48:15.7310678Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-vcw9kpw6mcqc'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T19:48:28.4804395Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e\", ActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", SubActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e ARID=9181dd6d-5c5a-470f-8a62-6f1f2abde29b > GW.Http.CallContext/9181dd6d-5c5a-470f-8a62-6f1f2abde29b)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x356d91091c60>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-13T19:48:15.7310676Z);\nlet _endTime = datetime(2026-10-14T19:48:15.7310678Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-vcw9kpw6mcqc'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T19:48:28.4804395Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e\", ActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", SubActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e ARID=9181dd6d-5c5a-470f-8a62-6f1f2abde29b > GW.Http.CallContext/9181dd6d-5c5a-470f-8a62-6f1f2abde29b)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x356d91091b60>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-13T19:48:15.7310676Z);\nlet _endTime = datetime(2026-10-14T19:48:15.7310678Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-vcw9kpw6mcqc'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T19:48:28.4804395Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e\", ActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", SubActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e ARID=9181dd6d-5c5a-470f-8a62-6f1f2abde29b > GW.Http.CallContext/9181dd6d-5c5a-470f-8a62-6f1f2abde29b)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x356d91091b20>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-13T19:48:15.7310676Z);\nlet _endTime = datetime(2026-10-14T19:48:15.7310678Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-vcw9kpw6mcqc'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T19:48:28.4804395Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e\", ActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", SubActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e ARID=9181dd6d-5c5a-470f-8a62-6f1f2abde29b > GW.Http.CallContext/9181dd6d-5c5a-470f-8a62-6f1f2abde29b)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x356d9135a600>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.007s.
Expected success, but got an error:
    <*fmt.wrapError | 0x356d91091ca0>: 
    failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-13T19:48:15.7310676Z);
    let _endTime = datetime(2026-10-14T19:48:15.7310678Z);
    // Extract cluster ID from CS logs using cid field
    let _cs_cluster_id = toscalar(
        database('ServiceLogs').table('clustersServiceLogs')
        | where timestamp between (_startTime .. _endTime)
        | where resource_id has 'kusto-logs-vcw9kpw6mcqc'
        | where isnotempty(log.cid)
        | distinct tostring(log.cid)
        | limit 1
    );
    // Find all Maestro resource IDs for this cluster
    let maestro_resource_ids = (
        database('ServiceLogs').table('containerLogs')
        | where timestamp between (_startTime .. _endTime)
        | where namespace_name has "maestro"
        | extend logStr = tostring(log)
        | where logStr has _cs_cluster_id
        | extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
        | where isnotempty(resourceId)
        | distinct resourceId
    );
    // Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
    union
        (
            // Maestro logs: Resource bundle operations for this cluster and its associated resources
            // Includes: resource creation, updates, deletions, status events, gRPC communications
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "maestro"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
            // Includes the CR reconciliation logs
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "hypershift"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        ),
        (
            // ACM Agent logs: ManifestWork reconciliation on the managed cluster
            // Includes: work agent operations, manifest application, status feedback
            database('ServiceLogs').table('containerLogs')
            | where timestamp between (_startTime .. _endTime)
            | where namespace_name has "open-cluster-management-agent"
            | extend logStr = tostring(log)
            | where logStr has _cs_cluster_id
            | project timestamp, container_name, msg = logStr, log
        )
    | order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-14T19:48:28.4804395Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e", ActivityId:"9181dd6d-5c5a-470f-8a62-6f1f2abde29b", SubActivityId:"9181dd6d-5c5a-470f-8a62-6f1f2abde29b", ActivityType:"GW.Http.CallContext", ParentActivityId:"9181dd6d-5c5a-470f-8a62-6f1f2abde29b", ActivityStack:"(Activity stack: CRID=KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e ARID=9181dd6d-5c5a-470f-8a62-6f1f2abde29b > GW.Http.CallContext/9181dd6d-5c5a-470f-8a62-6f1f2abde29b)"}, IsPermanent:false})
    {
        msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-13T19:48:15.7310676Z);\nlet _endTime = datetime(2026-10-14T19:48:15.7310678Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-vcw9kpw6mcqc'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T19:48:28.4804395Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e\", ActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", SubActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e ARID=9181dd6d-5c5a-470f-8a62-6f1f2abde29b > GW.Http.CallContext/9181dd6d-5c5a-470f-8a62-6f1f2abde29b)\"}, IsPermanent:false})",
        err: <*errors.joinError | 0x356d921ef170>{
            errs: [
                <*fmt.wrapError | 0x356d91091c80>{
                    msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-13T19:48:15.7310676Z);\nlet _endTime = datetime(2026-10-14T19:48:15.7310678Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-vcw9kpw6mcqc'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T19:48:28.4804395Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e\", ActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", SubActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e ARID=9181dd6d-5c5a-470f-8a62-6f1f2abde29b > GW.Http.CallContext/9181dd6d-5c5a-470f-8a62-6f1f2abde29b)\"}, IsPermanent:false})",
                    err: <*fmt.wrapError | 0x356d91091c60>{
                        msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-13T19:48:15.7310676Z);\nlet _endTime = datetime(2026-10-14T19:48:15.7310678Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-vcw9kpw6mcqc'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T19:48:28.4804395Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e\", ActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", SubActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e ARID=9181dd6d-5c5a-470f-8a62-6f1f2abde29b > GW.Http.CallContext/9181dd6d-5c5a-470f-8a62-6f1f2abde29b)\"}, IsPermanent:false})",
                        err: <*fmt.wrapError | 0x356d91091b60>{
                            msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-13T19:48:15.7310676Z);\nlet _endTime = datetime(2026-10-14T19:48:15.7310678Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-vcw9kpw6mcqc'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T19:48:28.4804395Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e\", ActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", SubActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e ARID=9181dd6d-5c5a-470f-8a62-6f1f2abde29b > GW.Http.CallContext/9181dd6d-5c5a-470f-8a62-6f1f2abde29b)\"}, IsPermanent:false})",
                            err: <*fmt.wrapError | 0x356d91091b20>{
                                msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-13T19:48:15.7310676Z);\nlet _endTime = datetime(2026-10-14T19:48:15.7310678Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-vcw9kpw6mcqc'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-14T19:48:28.4804395Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e\", ActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", SubActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"9181dd6d-5c5a-470f-8a62-6f1f2abde29b\", ActivityStack:\"(Activity stack: CRID=KGC.execute;7eea97f8-e218-4155-b4d6-a3666cff474e ARID=9181dd6d-5c5a-470f-8a62-6f1f2abde29b > GW.Http.CallContext/9181dd6d-5c5a-470f-8a62-6f1f2abde29b)\"}, IsPermanent:false})",
                                err: <*v2.OneApiError | 0x356d9135a600>{
                                    ErrorMessage: {
                                        Code: "LimitsExceeded",
                                        Message: "Request is invalid and cannot be executed.",
                                        Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
                                        Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
                                        Context: {
                                            Timestamp: ...,
                                            ServiceAlias: ...,
                                            MachineName: ...,
                                            ProcessName: ...,
                                            ProcessId: ...,
                                            ThreadId: ...,
                                            ClientRequestId: ...,
                                            ActivityId: ...,
                                            SubActivityId: ...,
                                            ActivityType: ...,
                                            ParentActivityId: ...,
                                            ActivityStack: ...,
                                        },
                                        IsPermanent: false,
                                    },
                                },
                            },
                        },
                    },
                },
            ],
        },
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74772110436010226618368
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRwaiting for a presented DNS-01 challenge for *
full failure pattern:
waiting for a presented DNS-01 challenge for *
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e10.84%1 of 119 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:444]: Timed out after 90.001s.
delegated NS and ACME TXT should resolve consistently on all authoritative nameservers, or the certificate should already be issued
Expected success, but got an error:
    <*errors.errorString | 0x2cc845a2b820>: 
    waiting for a presented DNS-01 challenge for *.apps.aro.ingress-cert-le.p7kd.j2563328.hcp.osadev.cloud
    {
        s: "waiting for a presented DNS-01 challenge for *.apps.aro.ingress-cert-le.p7kd.j2563328.hcp.osadev.cloud",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:444]: Timed out after 90.001s.
delegated NS and ACME TXT should resolve consistently on all authoritative nameservers, or the certificate should already be issued
Expected success, but got an error:
    <*errors.errorString | 0x2cc845a2b820>: 
    waiting for a presented DNS-01 challenge for *.apps.aro.ingress-cert-le.p7kd.j2563328.hcp.osadev.cloud
    {
        s: "waiting for a presented DNS-01 challenge for *.apps.aro.ingress-cert-le.p7kd.j2563328.hcp.osadev.cloud",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74162110506304622563328
alert [svc] BackendControllerQueueDepthHigh fired
full failure pattern:
alert [svc] BackendControllerQueueDepthHigh fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 6 days
alert1210.08%12 of 119 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 6 daysShow trend detailsOct 8: 19 · Oct 9: 4 · Oct 10: 0 · Oct 11: 2 · Oct 12: 8 · Oct 13: 9 · Oct 14: 12none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-15T00:13:57Z
  Ended:       2026-10-15T00:19:58Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5903232-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-15T00:02:33Z
  Ended:       2026-10-15T00:17:34Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8037120-svc", component="backend", name="csstatedump", region="westus3", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-10-15T00:02:33Z
  Ended:       2026-10-15T00:19:35Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8037120-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 3:
  State:       Resolved
  Started:     2026-10-15T00:03:34Z
  Ended:       2026-10-15T00:15:37Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8037120-svc", component="backend", name="clusterclusterserviceupdatedispatch", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterclusterserviceupdatedispatch has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-14T23:38:40Z
  Ended:       2026-10-14T23:56:38Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j0374272-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire12
Affected runs (12)
Date (UTC)Associated PRProw job
PR #74512110505977575903232
PR #75262110504372508037120
PR #66882110500986920374272
PR #75282110483351910486016
PR #75242110483335657558016
PR #68662110463708953055232
PR #75022110456066088112128
PR #75292110412598238253056
PR #74282110376503379234816
PR #75222110350863582105600
PR #73832110303886584582144
PR #75162110265029910597632
alert [hcp] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [hcp] KubeDeploymentReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
alert119.24%11 of 119 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsOct 8: 7 · Oct 9: 6 · Oct 10: 0 · Oct 11: 0 · Oct 12: 7 · Oct 13: 6 · Oct 14: 11none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-14T23:39:28Z
  Ended:       2026-10-14T23:53:28Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0214528-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.21:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0214528/providers/microsoft.monitor/accounts/hcps-j0214528", namespace="ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u", pod="arohcp-monitor-kube-state-metrics-7988dc648c-r2hbm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-14T23:39:28Z
  Ended:       2026-10-14T23:53:29Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0214528-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.21:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0214528/providers/microsoft.monitor/accounts/hcps-j0214528", namespace="ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u", pod="arohcp-monitor-kube-state-metrics-7988dc648c-r2hbm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-14T20:44:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6397952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6397952/providers/microsoft.monitor/accounts/hcps-j6397952", namespace="ocm-arohcpci01-2teosscr06nia6i1ff9khrnv5kp7a288-t4e9t7g2o3k6j4x", pod="arohcp-monitor-kube-state-metrics-7988dc648c-cl786", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2teosscr06nia6i1ff9khrnv5kp7a288-t4e9t7g2o3k6j4x/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-14T20:44:23Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6397952-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6397952/providers/microsoft.monitor/accounts/hcps-j6397952", namespace="ocm-arohcpci01-2teosscr06nia6i1ff9khrnv5kp7a288-t4e9t7g2o3k6j4x", pod="arohcp-monitor-kube-state-metrics-7988dc648c-cl786", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2teosscr06nia6i1ff9khrnv5kp7a288-t4e9t7g2o3k6j4x/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-14T20:01:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2922112-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.238:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2922112/providers/microsoft.monitor/accounts/hcps-j2922112", namespace="ocm-arohcpci01-2teodb2c2csmhm2lduce3cgjjgh1cnh8-n9y2x5u9j4c0t2r", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2wm6l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2teodb2c2csmhm2lduce3cgjjgh1cnh8-n9y2x5u9j4c0t2r/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-14T20:01:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2922112-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.238:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2922112/providers/microsoft.monitor/accounts/hcps-j2922112", namespace="ocm-arohcpci01-2teodb2c2csmhm2lduce3cgjjgh1cnh8-n9y2x5u9j4c0t2r", pod="arohcp-monitor-kube-state-metrics-7988dc648c-2wm6l", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2teodb2c2csmhm2lduce3cgjjgh1cnh8-n9y2x5u9j4c0t2r/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire11
Affected runs (11)
Date (UTC)Associated PRProw job
PR #63642110494690280214528
PR #66352110445968326397952
PR #75242110437162552922112
PR #73012110412681776205824
PR #74282110376503379234816
PR #74772110340691224694784
PR #74922110334900086247424
PR #75002110291728471691264
PR #67462110269833781186560
PR #75162110265029910597632
PR #67122110221944149250048
alert [svc] LeaderElectionLeaseStale fired
full failure pattern:
alert [svc] LeaderElectionLeaseStale fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
alert86.72%8 of 119 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsOct 8: 6 · Oct 9: 2 · Oct 10: 1 · Oct 11: 1 · Oct 12: 5 · Oct 13: 13 · Oct 14: 8none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-15T00:52:07Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j2275200-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.169:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2275200/providers/microsoft.monitor/accounts/services-j2275200", namespace="klusterlet-2tesd0o3f60ell7kn84k7na2ifcblhbl", pod="arohcp-monitor-kube-state-metrics-7988dc648c-wclqz", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tesd0o3f60ell7kn84k7na2ifcblhbl on cluster ci01-j2275200-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-14T22:51:50Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j6056960-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.176:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6056960/providers/microsoft.monitor/accounts/services-j6056960", namespace="klusterlet-2teqmhjn22ojhoa0ihg0gcmdrkaa544t", pod="arohcp-monitor-kube-state-metrics-7988dc648c-lrm27", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2teqmhjn22ojhoa0ihg0gcmdrkaa544t on cluster ci01-j6056960-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-14T21:31:37Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j2147200-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.205:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2147200/providers/microsoft.monitor/accounts/services-j2147200", namespace="klusterlet-2tepk3mcjhubmd4d9nfbho173hml93jh", pod="arohcp-monitor-kube-state-metrics-7988dc648c-jk4qm", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tepk3mcjhubmd4d9nfbho173hml93jh on cluster ci01-j2147200-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert LeaderElectionLeaseStale does not fire8
Affected runs (8)
Date (UTC)Associated PRProw job
PR #75402110508385492275200
PR #75382110474243606056960
PR #74972110452322852147200
PR #75242110437162552922112
PR #75272110408382648160256
PR #75252110338041116626944
PR #74992110304557253791744
PR #74412110272422715330560
cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
full failure pattern:
cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
e2e75.88%7 of 119 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsOct 8: 7 · Oct 9: 4 · Oct 10: 0 · Oct 11: 0 · Oct 12: 5 · Oct 13: 6 · Oct 14: 7none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-44v5fj7gn2vn/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-44v5fj7gn2vn/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-tp82mqtbfnjs/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-tp82mqtbfnjs/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-2bxwcdlh6wnp/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-2bxwcdlh6wnp/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation7
Affected runs (7)
Date (UTC)Associated PRProw job
PR #75442110520946510008320
PR #75402110508385492275200
PR #74922110490547654234112
PR #75002110452284272939008
PR #75242110407002151718912
PR #75042110213970672488448
PR #69682110191032388816896
alert [hcp] KubePodNotReady fired
full failure pattern:
alert [hcp] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 7 days
alert43.36%4 of 119 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 7 daysShow trend detailsOct 8: 3 · Oct 9: 2 · Oct 10: 1 · Oct 11: 4 · Oct 12: 5 · Oct 13: 6 · Oct 14: 4none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-14T20:10:46Z
  Ended:       2026-10-14T20:41:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3670144-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tep2lb9i2eso8etu810kd5rk9tkuprr-a1f8l5v2m8b4i0a", pod="router-84bfd8949d-rj9jm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tep2lb9i2eso8etu810kd5rk9tkuprr-a1f8l5v2m8b4i0a/router-84bfd8949d-rj9jm has been in a non-ready state for longer than 5 minutes.
alert fired 6 time(s) (3 unknown, 3 known)
Firing 1:
  State:       Resolved
  Started:     2026-10-14T14:05:08Z
  Ended:       2026-10-14T14:16:07Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2105600-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tejjhusc5envac7crsmjjhhgio0a4aj-k5o0u8o1m0i8z8z", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tejjhusc5envac7crsmjjhhgio0a4aj-k5o0u8o1m0i8z8z/etcd-1 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-14T14:05:09Z
  Ended:       2026-10-14T14:16:07Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2105600-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tejjc08ajbv33a72bs84k5higrqkmc1-private-kas", pod="etcd-2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tejjc08ajbv33a72bs84k5higrqkmc1-private-kas/etcd-2 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-14T14:05:09Z
  Ended:       2026-10-14T14:16:07Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2105600-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tejjl9d0k8tfj5mg7df7i9dvhecig15-d7v8f1i0h1a6r7x", pod="etcd-1", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tejjl9d0k8tfj5mg7df7i9dvhecig15-d7v8f1i0h1a6r7x/etcd-1 has been in a non-ready state for longer than 5 minutes.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-14T07:29:03Z
  Ended:       2026-10-14T07:43:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3579520-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tednh1ctcvl912rk2fj6486rd5emipo-j8c3b0w9t8v9t1z", pod="router-6fb85f79bd-67zsz", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2tednh1ctcvl912rk2fj6486rd5emipo-j8c3b0w9t8v9t1z/router-6fb85f79bd-67zsz has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #75292110447856593670144
PR #75222110350863582105600
PR #75042110251822873579520
PR #74952110199338117369856
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days
e2e32.52%3 of 119 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 5 daysShow trend detailsOct 8: 1 · Oct 9: 1 · Oct 10: 0 · Oct 11: 0 · Oct 12: 3 · Oct 13: 6 · Oct 14: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0x24524eb9ea50>: 
    GET https://rp.j0214528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/812fe66b-c9fc-46f9-8d46-bf8a0cd5068b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/812fe66b-c9fc-46f9-8d46-bf8a0cd5068b",
      "name": "812fe66b-c9fc-46f9-8d46-bf8a0cd5068b",
      "status": "Failed",
      "startTime": "2026-10-14T23:04:33.551295422Z",
      "endTime": "2026-10-14T23:33:41.530060174Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0214528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/812fe66b-c9fc-46f9-8d46-bf8a0cd5068b
            correlation ID: d14d5694-03de-4ac8-b587-4d0e35762d84
            client request ID: 5cfd8941-3a60-4ebe-8bc6-a659901f1339
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0x24524eb9ea50>: 
    GET https://rp.j0214528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/812fe66b-c9fc-46f9-8d46-bf8a0cd5068b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/812fe66b-c9fc-46f9-8d46-bf8a0cd5068b",
      "name": "812fe66b-c9fc-46f9-8d46-bf8a0cd5068b",
      "status": "Failed",
      "startTime": "2026-10-14T23:04:33.551295422Z",
      "endTime": "2026-10-14T23:33:41.530060174Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0214528.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/812fe66b-c9fc-46f9-8d46-bf8a0cd5068b
            correlation ID: d14d5694-03de-4ac8-b587-4d0e35762d84
            client request ID: 5cfd8941-3a60-4ebe-8bc6-a659901f1339
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0x225a34843540>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-5rp2rtlm9fj9" to finish creating: GET https://rp.j1691264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/680a3372-b50b-4c21-a07b-0256c4145619
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/680a3372-b50b-4c21-a07b-0256c4145619",
      "name": "680a3372-b50b-4c21-a07b-0256c4145619",
      "status": "Failed",
      "startTime": "2026-10-14T09:49:54.805487688Z",
      "endTime": "2026-10-14T10:18:57.194996434Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1691264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/680a3372-b50b-4c21-a07b-0256c4145619
            correlation ID: 5895723a-3ff0-431a-a927-5b6e194fc741
            client request ID: 350b7a37-c335-451c-b2e9-99a3b665afa1
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0x225a34843540>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-5rp2rtlm9fj9" to finish creating: GET https://rp.j1691264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/680a3372-b50b-4c21-a07b-0256c4145619
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/680a3372-b50b-4c21-a07b-0256c4145619",
      "name": "680a3372-b50b-4c21-a07b-0256c4145619",
      "status": "Failed",
      "startTime": "2026-10-14T09:49:54.805487688Z",
      "endTime": "2026-10-14T10:18:57.194996434Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1691264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/680a3372-b50b-4c21-a07b-0256c4145619
            correlation ID: 5895723a-3ff0-431a-a927-5b6e194fc741
            client request ID: 350b7a37-c335-451c-b2e9-99a3b665afa1
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0x3e322e6e67c0>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-7rswnntg2mwm" to finish creating: GET https://rp.j2666368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eeb29541-e4cc-419e-8a9e-17c262940b71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eeb29541-e4cc-419e-8a9e-17c262940b71",
      "name": "eeb29541-e4cc-419e-8a9e-17c262940b71",
      "status": "Failed",
      "startTime": "2026-10-14T05:02:26.935820049Z",
      "endTime": "2026-10-14T05:31:33.678218143Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2666368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eeb29541-e4cc-419e-8a9e-17c262940b71
            correlation ID: 91b1a5de-a9bc-4e6e-ad6d-e45a7cb1831a
            client request ID: 4a6ae7e3-e39d-4cda-bb39-3082e7e48f98
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0x3e322e6e67c0>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-7rswnntg2mwm" to finish creating: GET https://rp.j2666368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eeb29541-e4cc-419e-8a9e-17c262940b71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eeb29541-e4cc-419e-8a9e-17c262940b71",
      "name": "eeb29541-e4cc-419e-8a9e-17c262940b71",
      "status": "Failed",
      "startTime": "2026-10-14T05:02:26.935820049Z",
      "endTime": "2026-10-14T05:31:33.678218143Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2666368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eeb29541-e4cc-419e-8a9e-17c262940b71
            correlation ID: 91b1a5de-a9bc-4e6e-ad6d-e45a7cb1831a
            client request ID: 4a6ae7e3-e39d-4cda-bb39-3082e7e48f98
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
Affected runs (3)
Date (UTC)Associated PRProw job
PR #63642110494690280214528
PR #75002110291728471691264
PR #75092110213919732666368
alert [hcp] KubeStatefulSetReplicasMismatch fired
full failure pattern:
alert [hcp] KubeStatefulSetReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days
alert21.68%2 of 119 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 4 daysShow trend detailsOct 8: 1 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 3 · Oct 13: 3 · Oct 14: 2none
Full failure examples (2)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-14T23:34:33Z
  Ended:       2026-10-14T23:51:31Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0214528-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.21:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0214528/providers/microsoft.monitor/accounts/hcps-j0214528", namespace="ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u", pod="arohcp-monitor-kube-state-metrics-7988dc648c-r2hbm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-14T23:35:30Z
  Ended:       2026-10-14T23:52:28Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0214528-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.21:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0214528/providers/microsoft.monitor/accounts/hcps-j0214528", namespace="ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u", pod="arohcp-monitor-kube-state-metrics-7988dc648c-r2hbm", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-14T10:09:45Z
  Ended:       2026-10-14T10:36:51Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1691264-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.235:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1691264/providers/microsoft.monitor/accounts/hcps-j1691264", namespace="ocm-arohcpci01-2teg4gob3eq3kct1j85ncs4avp9immt5-r0t7q6r8g7p8n9p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-9qldk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2teg4gob3eq3kct1j85ncs4avp9immt5-r0t7q6r8g7p8n9p/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-14T10:09:45Z
  Ended:       2026-10-14T10:36:51Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j1691264-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.235:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1691264/providers/microsoft.monitor/accounts/hcps-j1691264", namespace="ocm-arohcpci01-2teg4gob3eq3kct1j85ncs4avp9immt5-r0t7q6r8g7p8n9p", pod="arohcp-monitor-kube-state-metrics-7988dc648c-9qldk", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2teg4gob3eq3kct1j85ncs4avp9immt5-r0t7q6r8g7p8n9p/etcd has not matched the expected number of replicas for longer than 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #63642110494690280214528
PR #75002110291728471691264
alert [hcp] KubePodCrashLooping fired
full failure pattern:
alert [hcp] KubePodCrashLooping fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
alert21.68%2 of 119 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsOct 8: 2 · Oct 9: 2 · Oct 10: 0 · Oct 11: 0 · Oct 12: 4 · Oct 13: 2 · Oct 14: 2none
Full failure examples (2)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-14T23:26:27Z
  Ended:       2026-10-14T23:56:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0214528-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.21:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0214528/providers/microsoft.monitor/accounts/hcps-j0214528", namespace="ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e7a6870f-14b4-45e0-9c19-fce7f03813dc"
  Description: Pod ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-14T23:26:27Z
  Ended:       2026-10-14T23:56:29Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0214528-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.21:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0214528/providers/microsoft.monitor/accounts/hcps-j0214528", namespace="ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e7a6870f-14b4-45e0-9c19-fce7f03813dc"
  Description: Pod ocm-arohcpci01-2terp08u81ovror5h3o9svngoo0a5t0t-q6f6k8k9x1s1l6u/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-14T10:12:41Z
  Ended:       2026-10-14T10:40:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1691264-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.235:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1691264/providers/microsoft.monitor/accounts/hcps-j1691264", namespace="ocm-arohcpci01-2teg4gob3eq3kct1j85ncs4avp9immt5-r0t7q6r8g7p8n9p", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="59d27077-4da5-493d-b6e6-b015a3abbc8a"
  Description: Pod ocm-arohcpci01-2teg4gob3eq3kct1j85ncs4avp9immt5-r0t7q6r8g7p8n9p/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-10-14T10:12:41Z
  Ended:       2026-10-14T10:41:47Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j1691264-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.235:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1691264/providers/microsoft.monitor/accounts/hcps-j1691264", namespace="ocm-arohcpci01-2teg4gob3eq3kct1j85ncs4avp9immt5-r0t7q6r8g7p8n9p", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="59d27077-4da5-493d-b6e6-b015a3abbc8a"
  Description: Pod ocm-arohcpci01-2teg4gob3eq3kct1j85ncs4avp9immt5-r0t7q6r8g7p8n9p/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #63642110494690280214528
PR #75002110291728471691264
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e21.68%2 of 119 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsOct 8: 2 · Oct 9: 2 · Oct 10: 0 · Oct 11: 0 · Oct 12: 2 · Oct 13: 0 · Oct 14: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
    <*errors.joinError | 0xba275880060>: 
    resource group "customer-rg-xns69mfwd2hq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-xns69mfwd2hq" to finish deleting: GET https://rp.j6205824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3176ad81-fee9-4cf2-8688-8a1c24844387
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3176ad81-fee9-4cf2-8688-8a1c24844387",
      "name": "3176ad81-fee9-4cf2-8688-8a1c24844387",
      "status": "Failed",
      "startTime": "2026-10-14T18:01:05.468018625Z",
      "endTime": "2026-10-14T18:25:10.474406797Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ten52g4l2k3e0ijcet6akeh14ne2nbv still exists (deletion dispatched at 2026-10-14T18:01:06Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6205824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3176ad81-fee9-4cf2-8688-8a1c24844387
            correlation ID: fdc30364-8ed7-4835-a286-f7a091278652
            client request ID: a6850f11-48fa-4616-8112-2b5b344c479a
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
    <*errors.joinError | 0xba275880060>: 
    resource group "customer-rg-xns69mfwd2hq": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-xns69mfwd2hq" to finish deleting: GET https://rp.j6205824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3176ad81-fee9-4cf2-8688-8a1c24844387
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3176ad81-fee9-4cf2-8688-8a1c24844387",
      "name": "3176ad81-fee9-4cf2-8688-8a1c24844387",
      "status": "Failed",
      "startTime": "2026-10-14T18:01:05.468018625Z",
      "endTime": "2026-10-14T18:25:10.474406797Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2ten52g4l2k3e0ijcet6akeh14ne2nbv still exists (deletion dispatched at 2026-10-14T18:01:06Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6205824.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3176ad81-fee9-4cf2-8688-8a1c24844387
            correlation ID: fdc30364-8ed7-4835-a286-f7a091278652
            client request ID: a6850f11-48fa-4616-8112-2b5b344c479a
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
    <*errors.joinError | 0x29ce27c860f0>: 
    resource group "ea-list-7knxqnrqn2r4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listrj8r2m" in resourcegroup="ea-list-7knxqnrqn2r4" to finish deleting: GET https://rp.j4694784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7d6add57-b9e0-49e9-82fe-b8f41bea59ca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7d6add57-b9e0-49e9-82fe-b8f41bea59ca",
      "name": "7d6add57-b9e0-49e9-82fe-b8f41bea59ca",
      "status": "Failed",
      "startTime": "2026-10-14T13:07:39.341755996Z",
      "endTime": "2026-10-14T13:31:39.648045292Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2teirqf5984h2i8hr97b4hmu155vmka1 still exists (deletion dispatched at 2026-10-14T13:07:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4694784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7d6add57-b9e0-49e9-82fe-b8f41bea59ca
            correlation ID: a29d8857-e9bf-4a4f-bec4-d3e72ff8ea78
            client request ID: 11bba4e8-0934-463b-98c2-cdf775749155
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
    <*errors.joinError | 0x29ce27c860f0>: 
    resource group "ea-list-7knxqnrqn2r4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listrj8r2m" in resourcegroup="ea-list-7knxqnrqn2r4" to finish deleting: GET https://rp.j4694784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7d6add57-b9e0-49e9-82fe-b8f41bea59ca
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7d6add57-b9e0-49e9-82fe-b8f41bea59ca",
      "name": "7d6add57-b9e0-49e9-82fe-b8f41bea59ca",
      "status": "Failed",
      "startTime": "2026-10-14T13:07:39.341755996Z",
      "endTime": "2026-10-14T13:31:39.648045292Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2teirqf5984h2i8hr97b4hmu155vmka1 still exists (deletion dispatched at 2026-10-14T13:07:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4694784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7d6add57-b9e0-49e9-82fe-b8f41bea59ca
            correlation ID: a29d8857-e9bf-4a4f-bec4-d3e72ff8ea78
            client request ID: 11bba4e8-0934-463b-98c2-cdf775749155
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #73012110412681776205824
PR #74772110340691224694784
alert [svc] MaestroAgentMetricsProxyDown fired
full failure pattern:
alert [svc] MaestroAgentMetricsProxyDown fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 4 days
alert21.68%2 of 119 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 4 daysShow trend detailsOct 8: 2 · Oct 9: 3 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 1 · Oct 14: 2none
Full failure examples (2)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-14T19:52:31Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j2981888-mgmt-2", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-14T16:54:29Z
  Ended:       2026-10-14T17:00:24Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j8174976-mgmt-1", component="monitoring-infrastructure", region="centralus", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert MaestroAgentMetricsProxyDown does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #69462110441834562981888
PR #75092110398398338174976
timeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days
e2e21.68%2 of 119 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 3 daysShow trend detailsOct 8: 0 · Oct 9: 2 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 0 · Oct 14: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x1399efb46de0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-2n7cffz9j9vb" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2n7cffz9j9vb, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-2n7cffz9j9vb\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2n7cffz9j9vb, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x1399ef809d30>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2n7cffz9j9vb",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x1399efb46de0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-2n7cffz9j9vb" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2n7cffz9j9vb, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-2n7cffz9j9vb\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2n7cffz9j9vb, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x1399ef809d30>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-2n7cffz9j9vb",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x14825bb062d0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-gvb49lpzrwdc" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-gvb49lpzrwdc, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-gvb49lpzrwdc\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-gvb49lpzrwdc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x14825b6451e0>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-gvb49lpzrwdc",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x14825bb062d0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-gvb49lpzrwdc" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-gvb49lpzrwdc, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-gvb49lpzrwdc\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-gvb49lpzrwdc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x14825b6451e0>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-gvb49lpzrwdc",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.202
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74772110436010226618368
PR #74772110340691224694784
alert [svc] FrontendPathLatency fired
full failure pattern:
alert [svc] FrontendPathLatency fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 4 days
alert10.84%1 of 119 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 4 daysShow trend detailsOct 8: 1 · Oct 9: 1 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 1 · Oct 14: 1none
Full failure examples (1)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-14T20:45:07Z
  Ended:       2026-10-14T20:56:08Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j2147200-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: More than 5% of frontend requests for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j2147200-svc).
Firing 2:
  State:       Resolved
  Started:     2026-10-14T21:06:15Z
  Ended:       2026-10-14T21:18:04Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j2147200-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: More than 5% of frontend requests for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j2147200-svc).
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert FrontendPathLatency does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74972110452322852147200
alert [svc] KubePodNotReady fired
full failure pattern:
alert [svc] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
alert10.84%1 of 119 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsOct 8: 1 · Oct 9: 1 · Oct 10: 0 · Oct 11: 0 · Oct 12: 2 · Oct 13: 1 · Oct 14: 1none
Full failure examples (1)
alert fired 70 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-4fnrs", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-4fnrs has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-qd4lc", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-qd4lc has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-q9htv", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-q9htv has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-2v2ll", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-2v2ll has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-p45vd", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-p45vd has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-v2bxl", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-v2bxl has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-rsvx7", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-rsvx7 has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-zw6fj", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-zw6fj has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-r5r7n", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-r5r7n has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-xb9hp", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-xb9hp has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-bvr8s", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-bvr8s has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-vsrr8", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-vsrr8 has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-srv48", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-srv48 has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-q5vds", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-q5vds has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-rjwtc", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-rjwtc has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-nzgpk", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-nzgpk has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-4slr2", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-4slr2 has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-m7ftv", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-m7ftv has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-v7hdm", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-v7hdm has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-86ssh", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-86ssh has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-jp6fj", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-jp6fj has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-xjxzz", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-xjxzz has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-b7ph9", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-b7ph9 has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-drpfk", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-drpfk has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-6q6w9", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-6q6w9 has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-77r8t", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-77r8t has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-vhdzq", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-vhdzq has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-hdcnl", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-hdcnl has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-5z589", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-5z589 has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-kr492", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-kr492 has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-bcl97", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-bcl97 has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-s49mm", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-s49mm has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-drmdx", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-drmdx has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-sm8gg", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-sm8gg has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-2xnc2", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-2xnc2 has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-7x9gw", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-7x9gw has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-bddnr", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-bddnr has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-kz42p", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-kz42p has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-nz88g", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-nz88g has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-4pq7x", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-4pq7x has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-flmn9", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-flmn9 has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-fwvds", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-fwvds has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-mk9q7", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-mk9q7 has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-6bqxw", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-6bqxw has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-xn8q4", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-xn8q4 has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-4s44z", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-4s44z has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-fhhv2", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-fhhv2 has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-t8v8f", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-t8v8f has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-2sg4p", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-2sg4p has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-mh74p", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-mh74p has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-fgn4b", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-fgn4b has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-lq8md", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-lq8md has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-wcdz6", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-wcdz6 has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-qbfw5", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-qbfw5 has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-nshz9", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-nshz9 has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-f6nsp", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-f6nsp has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-z6t2j", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-z6t2j has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-s75xf", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-s75xf has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-bmx5q", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-bmx5q has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-t57qd", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-t57qd has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-4ntcs", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-4ntcs has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-cjtgq", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-cjtgq has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-6jl7m", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-6jl7m has been in a non-ready state for longer than 5 minutes.
Firing 64:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-79sxk", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-79sxk has been in a non-ready state for longer than 5 minutes.
Firing 65:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-pqnwt", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-pqnwt has been in a non-ready state for longer than 5 minutes.
Firing 66:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-hplkm", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-hplkm has been in a non-ready state for longer than 5 minutes.
Firing 67:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-drfrc", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-drfrc has been in a non-ready state for longer than 5 minutes.
Firing 68:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-9k4dd", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-9k4dd has been in a non-ready state for longer than 5 minutes.
Firing 69:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-4zv7k", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-4zv7k has been in a non-ready state for longer than 5 minutes.
Firing 70:
  State:       Fired (not resolved)
  Started:     2026-10-14T17:56:11Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6227072-svc", component="kubernetes-infrastructure", namespace="monitoring", pod="kube-events-5ddf5fbfd4-ql2fg", region="centralus", severity="warning"
  Description: Pod monitoring/kube-events-5ddf5fbfd4-ql2fg has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubePodNotReady does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75192110417722436227072
ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 5 days
e2e10.84%1 of 119 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 5 daysShow trend detailsOct 8: 1 · Oct 9: 0 · Oct 10: 0 · Oct 11: 1 · Oct 12: 1 · Oct 13: 1 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0x12acc23ba300>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-kwgn8mlj4cqw" to finish creating: GET https://rp.j2922112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/431cc2e5-1210-4f39-b481-aac97e02bf64
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/431cc2e5-1210-4f39-b481-aac97e02bf64' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/431cc2e5-1210-4f39-b481-aac97e02bf64"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2922112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/431cc2e5-1210-4f39-b481-aac97e02bf64
            correlation ID: 6cf5166e-d7f7-4d42-8f52-96bfabf5f7df
            client request ID: d4b2a1df-961f-48a7-9e13-886e82b9c210
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0x12acc23ba300>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-kwgn8mlj4cqw" to finish creating: GET https://rp.j2922112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/431cc2e5-1210-4f39-b481-aac97e02bf64
    --------------------------------------------------------------------------------
    RESPONSE 404: 404 Not Found
    ERROR CODE: ResourceNotFound
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "ResourceNotFound",
        "message": "The resource 'locations/hcpOperationStatuses/431cc2e5-1210-4f39-b481-aac97e02bf64' under resource group '' was not found.",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/431cc2e5-1210-4f39-b481-aac97e02bf64"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2922112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/431cc2e5-1210-4f39-b481-aac97e02bf64
            correlation ID: 6cf5166e-d7f7-4d42-8f52-96bfabf5f7df
            client request ID: d4b2a1df-961f-48a7-9e13-886e82b9c210
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75242110437162552922112
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>
full failure pattern:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url>
Signal: FlakeSignal: Flake — present in 4 prior week(s); spread across 3 days
e2e10.84%1 of 119 job runs affectedFlakeSignal: Flake — present in 4 prior week(s); spread across 3 daysShow trend detailsOct 8: 1 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 2 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:364]: Timed out after 300.007s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0x3db417569280>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:54452->172.30.0.10:53: read: connection refused
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:54452->172.30.0.10:53: read: connection refused",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:364]: Timed out after 300.007s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0x3db417569280>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:54452->172.30.0.10:53: read: connection refused
    {
        s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:54452->172.30.0.10:53: read: connection refused",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75242110483335657558016
| take 101, Post "<url>": context deadline exceeded
full failure pattern:
| take 101, Post "<url>": context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e2621.85%26 of 119 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 2 · Oct 14: 26none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dp2n7z9g2mfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xb36d0212940>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dp2n7z9g2mfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dp2n7z9g2mfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-15T00:30:08.2263967Z);
    let desires = cosmosResourceSnapshots
    | where resourceID startswith strcat(target, "/")
    | where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
    | extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
    | where isempty(content.deletionTimestamp)
    | extend manifest = content.properties.status.kubeContent
    | where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
    | extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
        managementCluster = tolower(tostring(content.properties.spec.managementCluster))
    | where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
    | project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
    let fleet = cosmosResourceSnapshots
    | where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
    | extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
    | where isempty(content.deletionTimestamp)
    | extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
    | where isnotempty(clusterName)
    | project environment, region, sourceCluster, managementCluster, cluster = clusterName;
    let hcp = desires
    | join kind=inner fleet on environment, region, sourceCluster, managementCluster
    | distinct environment, region, cluster, namespace;
    let inventory = materialize(kubernetesResourceSnapshots
    | where timestamp >= start and objectKind in ("Pod", "Deployment")
    | join kind=leftsemi hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
    | where event != "Delete" and isempty(object.metadata.deletionTimestamp));
    let pods = inventory
    | where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
    | project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
    let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
    let deployments = inventory
    | where objectKind == "Deployment" and name == "router"
    | project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
    let summaries = containerLogs
    | where timestamp >= start and container_name == "swift-recorder"
    | where log.controller_name == "swift-router-check" and log.field == "summary"
    | extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
    | join kind=leftsemi hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
    | project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
        SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
    hcp
    | join kind=leftouter deployments on environment, region, cluster, namespace
    | join kind=leftouter counts on environment, region, cluster, namespace
    | join kind=leftouter pods on environment, region, cluster, namespace
    | join kind=leftouter summaries on environment, region, cluster, namespace, UID
    | project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
        Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
    | order by Scope asc, UID asc
    | take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dp2n7z9g2mfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dp2n7z9g2mfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-15T00:30:08.2263967Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
        err: <*fmt.wrapError | 0xb36d0212920>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dp2n7z9g2mfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-15T00:30:08.2263967Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
            err: <*errors.Error | 0xb36cf27e550>{
                Op: 1,
                Kind: 7,
                Err: <*fmt.wrapError | 0xb36d0212900>{
                    msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dp2n7z9g2mfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-15T00:30:08.2263967Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                    err: <*url.Error | 0xb36d137e150>{
                        Op: "Post",
                        URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                        Err: <context.deadlineExceededError>{},
                    },
                },
                restErrMsg: nil,
                decoded: nil,
                permanent: false,
                inner: nil,
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dp2n7z9g2mfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xb36d0212940>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dp2n7z9g2mfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dp2n7z9g2mfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-15T00:30:08.2263967Z);
    let desires = cosmosResourceSnapshots
    | where resourceID startswith strcat(target, "/")
    | where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
    | extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
    | where isempty(content.deletionTimestamp)
    | extend manifest = content.properties.status.kubeContent
    | where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
    | extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
        managementCluster = tolower(tostring(content.properties.spec.managementCluster))
    | where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
    | project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
    let fleet = cosmosResourceSnapshots
    | where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
    | extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
    | where isempty(content.deletionTimestamp)
    | extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
    | where isnotempty(clusterName)
    | project environment, region, sourceCluster, managementCluster, cluster = clusterName;
    let hcp = desires
    | join kind=inner fleet on environment, region, sourceCluster, managementCluster
    | distinct environment, region, cluster, namespace;
    let inventory = materialize(kubernetesResourceSnapshots
    | where timestamp >= start and objectKind in ("Pod", "Deployment")
    | join kind=leftsemi hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
    | where event != "Delete" and isempty(object.metadata.deletionTimestamp));
    let pods = inventory
    | where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
    | project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
    let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
    let deployments = inventory
    | where objectKind == "Deployment" and name == "router"
    | project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
    let summaries = containerLogs
    | where timestamp >= start and container_name == "swift-recorder"
    | where log.controller_name == "swift-router-check" and log.field == "summary"
    | extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
    | join kind=leftsemi hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
    | project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
        SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
    hcp
    | join kind=leftouter deployments on environment, region, cluster, namespace
    | join kind=leftouter counts on environment, region, cluster, namespace
    | join kind=leftouter pods on environment, region, cluster, namespace
    | join kind=leftouter summaries on environment, region, cluster, namespace, UID
    | project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
        Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
    | order by Scope asc, UID asc
    | take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dp2n7z9g2mfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dp2n7z9g2mfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-15T00:30:08.2263967Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
        err: <*fmt.wrapError | 0xb36d0212920>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dp2n7z9g2mfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-15T00:30:08.2263967Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
            err: <*errors.Error | 0xb36cf27e550>{
                Op: 1,
                Kind: 7,
                Err: <*fmt.wrapError | 0xb36d0212900>{
                    msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-dp2n7z9g2mfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-15T00:30:08.2263967Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                    err: <*url.Error | 0xb36d137e150>{
                        Op: "Post",
                        URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                        Err: <context.deadlineExceededError>{},
                    },
                },
                restErrMsg: nil,
                decoded: nil,
                permanent: false,
                inner: nil,
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-gmq4jh246r5n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1cdf5a79e560>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-gmq4jh246r5n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-gmq4jh246r5n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-14T23:20:39.7053116Z);
    let desires = cosmosResourceSnapshots
    | where resourceID startswith strcat(target, "/")
    | where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
    | extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
    | where isempty(content.deletionTimestamp)
    | extend manifest = content.properties.status.kubeContent
    | where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
    | extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
        managementCluster = tolower(tostring(content.properties.spec.managementCluster))
    | where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
    | project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
    let fleet = cosmosResourceSnapshots
    | where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
    | extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
    | where isempty(content.deletionTimestamp)
    | extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
    | where isnotempty(clusterName)
    | project environment, region, sourceCluster, managementCluster, cluster = clusterName;
    let hcp = desires
    | join kind=inner fleet on environment, region, sourceCluster, managementCluster
    | distinct environment, region, cluster, namespace;
    let inventory = materialize(kubernetesResourceSnapshots
    | where timestamp >= start and objectKind in ("Pod", "Deployment")
    | join kind=leftsemi hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
    | where event != "Delete" and isempty(object.metadata.deletionTimestamp));
    let pods = inventory
    | where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
    | project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
    let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
    let deployments = inventory
    | where objectKind == "Deployment" and name == "router"
    | project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
    let summaries = containerLogs
    | where timestamp >= start and container_name == "swift-recorder"
    | where log.controller_name == "swift-router-check" and log.field == "summary"
    | extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
    | join kind=leftsemi hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
    | project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
        SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
    hcp
    | join kind=leftouter deployments on environment, region, cluster, namespace
    | join kind=leftouter counts on environment, region, cluster, namespace
    | join kind=leftouter pods on environment, region, cluster, namespace
    | join kind=leftouter summaries on environment, region, cluster, namespace, UID
    | project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
        Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
    | order by Scope asc, UID asc
    | take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-gmq4jh246r5n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-gmq4jh246r5n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-14T23:20:39.7053116Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
        err: <*fmt.wrapError | 0x1cdf5a79e520>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-gmq4jh246r5n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-14T23:20:39.7053116Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
            err: <*errors.Error | 0x1cdf5a5c4500>{
                Op: 1,
                Kind: 7,
                Err: <*fmt.wrapError | 0x1cdf5a79e500>{
                    msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-gmq4jh246r5n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-14T23:20:39.7053116Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                    err: <*url.Error | 0x1cdf5bbab080>{
                        Op: "Post",
                        URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                        Err: <context.deadlineExceededError>{},
                    },
                },
                restErrMsg: nil,
                decoded: nil,
                permanent: false,
                inner: nil,
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-gmq4jh246r5n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1cdf5a79e560>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-gmq4jh246r5n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-gmq4jh246r5n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-14T23:20:39.7053116Z);
    let desires = cosmosResourceSnapshots
    | where resourceID startswith strcat(target, "/")
    | where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
    | extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
    | where isempty(content.deletionTimestamp)
    | extend manifest = content.properties.status.kubeContent
    | where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
    | extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
        managementCluster = tolower(tostring(content.properties.spec.managementCluster))
    | where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
    | project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
    let fleet = cosmosResourceSnapshots
    | where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
    | extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
    | where isempty(content.deletionTimestamp)
    | extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
    | where isnotempty(clusterName)
    | project environment, region, sourceCluster, managementCluster, cluster = clusterName;
    let hcp = desires
    | join kind=inner fleet on environment, region, sourceCluster, managementCluster
    | distinct environment, region, cluster, namespace;
    let inventory = materialize(kubernetesResourceSnapshots
    | where timestamp >= start and objectKind in ("Pod", "Deployment")
    | join kind=leftsemi hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
    | where event != "Delete" and isempty(object.metadata.deletionTimestamp));
    let pods = inventory
    | where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
    | project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
    let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
    let deployments = inventory
    | where objectKind == "Deployment" and name == "router"
    | project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
    let summaries = containerLogs
    | where timestamp >= start and container_name == "swift-recorder"
    | where log.controller_name == "swift-router-check" and log.field == "summary"
    | extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
    | join kind=leftsemi hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
    | project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
        SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
    hcp
    | join kind=leftouter deployments on environment, region, cluster, namespace
    | join kind=leftouter counts on environment, region, cluster, namespace
    | join kind=leftouter pods on environment, region, cluster, namespace
    | join kind=leftouter summaries on environment, region, cluster, namespace, UID
    | project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
        Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
    | order by Scope asc, UID asc
    | take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-gmq4jh246r5n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-gmq4jh246r5n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-14T23:20:39.7053116Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
        err: <*fmt.wrapError | 0x1cdf5a79e520>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-gmq4jh246r5n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-14T23:20:39.7053116Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
            err: <*errors.Error | 0x1cdf5a5c4500>{
                Op: 1,
                Kind: 7,
                Err: <*fmt.wrapError | 0x1cdf5a79e500>{
                    msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-gmq4jh246r5n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-14T23:20:39.7053116Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                    err: <*url.Error | 0x1cdf5bbab080>{
                        Op: "Post",
                        URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                        Err: <context.deadlineExceededError>{},
                    },
                },
                restErrMsg: nil,
                decoded: nil,
                permanent: false,
                inner: nil,
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spmr2bkfrwww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1b5b518622a0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spmr2bkfrwww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spmr2bkfrwww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-14T23:28:59.0626387Z);
    let desires = cosmosResourceSnapshots
    | where resourceID startswith strcat(target, "/")
    | where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
    | extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
    | where isempty(content.deletionTimestamp)
    | extend manifest = content.properties.status.kubeContent
    | where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
    | extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
        managementCluster = tolower(tostring(content.properties.spec.managementCluster))
    | where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
    | project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
    let fleet = cosmosResourceSnapshots
    | where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
    | extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
    | where isempty(content.deletionTimestamp)
    | extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
    | where isnotempty(clusterName)
    | project environment, region, sourceCluster, managementCluster, cluster = clusterName;
    let hcp = desires
    | join kind=inner fleet on environment, region, sourceCluster, managementCluster
    | distinct environment, region, cluster, namespace;
    let inventory = materialize(kubernetesResourceSnapshots
    | where timestamp >= start and objectKind in ("Pod", "Deployment")
    | join kind=leftsemi hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
    | where event != "Delete" and isempty(object.metadata.deletionTimestamp));
    let pods = inventory
    | where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
    | project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
    let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
    let deployments = inventory
    | where objectKind == "Deployment" and name == "router"
    | project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
    let summaries = containerLogs
    | where timestamp >= start and container_name == "swift-recorder"
    | where log.controller_name == "swift-router-check" and log.field == "summary"
    | extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
    | join kind=leftsemi hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
    | project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
        SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
    hcp
    | join kind=leftouter deployments on environment, region, cluster, namespace
    | join kind=leftouter counts on environment, region, cluster, namespace
    | join kind=leftouter pods on environment, region, cluster, namespace
    | join kind=leftouter summaries on environment, region, cluster, namespace, UID
    | project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
        Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
    | order by Scope asc, UID asc
    | take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spmr2bkfrwww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spmr2bkfrwww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-14T23:28:59.0626387Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
        err: <*fmt.wrapError | 0x1b5b51862280>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spmr2bkfrwww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-14T23:28:59.0626387Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
            err: <*errors.Error | 0x1b5b50bb8140>{
                Op: 1,
                Kind: 7,
                Err: <*fmt.wrapError | 0x1b5b51862260>{
                    msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spmr2bkfrwww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-14T23:28:59.0626387Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                    err: <*url.Error | 0x1b5b528f0810>{
                        Op: "Post",
                        URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                        Err: <context.deadlineExceededError>{},
                    },
                },
                restErrMsg: nil,
                decoded: nil,
                permanent: false,
                inner: nil,
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spmr2bkfrwww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x1b5b518622a0>: 
    VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spmr2bkfrwww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spmr2bkfrwww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-14T23:28:59.0626387Z);
    let desires = cosmosResourceSnapshots
    | where resourceID startswith strcat(target, "/")
    | where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
    | extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
    | where isempty(content.deletionTimestamp)
    | extend manifest = content.properties.status.kubeContent
    | where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
    | extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
        managementCluster = tolower(tostring(content.properties.spec.managementCluster))
    | where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
    | project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
    let fleet = cosmosResourceSnapshots
    | where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
    | extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
    | join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
    | summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
    | summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
    | where isempty(content.deletionTimestamp)
    | extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
    | where isnotempty(clusterName)
    | project environment, region, sourceCluster, managementCluster, cluster = clusterName;
    let hcp = desires
    | join kind=inner fleet on environment, region, sourceCluster, managementCluster
    | distinct environment, region, cluster, namespace;
    let inventory = materialize(kubernetesResourceSnapshots
    | where timestamp >= start and objectKind in ("Pod", "Deployment")
    | join kind=leftsemi hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
    | where event != "Delete" and isempty(object.metadata.deletionTimestamp));
    let pods = inventory
    | where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
    | project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
    let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
    let deployments = inventory
    | where objectKind == "Deployment" and name == "router"
    | project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
    let summaries = containerLogs
    | where timestamp >= start and container_name == "swift-recorder"
    | where log.controller_name == "swift-router-check" and log.field == "summary"
    | extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
    | join kind=leftsemi hcp on environment, region, cluster, namespace
    | summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
    | project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
        SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
    hcp
    | join kind=leftouter deployments on environment, region, cluster, namespace
    | join kind=leftouter counts on environment, region, cluster, namespace
    | join kind=leftouter pods on environment, region, cluster, namespace
    | join kind=leftouter summaries on environment, region, cluster, namespace, UID
    | project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
        Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
    | order by Scope asc, UID asc
    | take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
    {
        msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spmr2bkfrwww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.002s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spmr2bkfrwww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-14T23:28:59.0626387Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
        err: <*fmt.wrapError | 0x1b5b51862280>{
            msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spmr2bkfrwww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-14T23:28:59.0626387Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
            err: <*errors.Error | 0x1b5b50bb8140>{
                Op: 1,
                Kind: 7,
                Err: <*fmt.wrapError | 0x1b5b51862260>{
                    msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-spmr2bkfrwww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-14T23:28:59.0626387Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n    managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n    SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n    Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
                    err: <*url.Error | 0x1b5b528f0810>{
                        Op: "Post",
                        URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
                        Err: <context.deadlineExceededError>{},
                    },
                },
                restErrMsg: nil,
                decoded: nil,
                permanent: false,
                inner: nil,
            },
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services26
Affected runs (26)
Date (UTC)Associated PRProw job
PR #75322110513999702396928
PR #66882110500986920374272
PR #73422110500816379973632
PR #74402110483415672295424
PR #75022110456066088112128
PR #74982110452354565279744
PR #74972110452322852147200
PR #75002110452284272939008
PR #74402110444073914470400
PR #75272110408382648160256
PR #75242110407002151718912
PR #75002110389003348873216
PR #75242110371287573467136
PR #67462110361307805913088
PR #74772110340691224694784
PR #75002110334902741241856
PR #74922110334900086247424
PR #75022110334374363795456
PR #74972110305876723109888
PR #67462110269833781186560
PR #75182110268638186442752
PR #74512110267140211740672
PR #75192110267078714855424
PR #74772110235250528358400
PR #75042110213970672488448
PR #72482110168566228586496
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...
full failure pattern:
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)
e2e54.20%5 of 119 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)Show trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 5none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x1fd36b60420>: 
    DNS for route host agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x1fd36b60400>{
            msg: "DNS for agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x1fd36b60420>: 
    DNS for route host agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x1fd36b60400>{
            msg: "DNS for agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mdh26.apps.aro.v20261001.d30u.j7447552.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x14a4f3922100>: 
    DNS for route host agnhost-e2e-sample-app-cfkmf.apps.aro.v20261001.113i.j5059456.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-cfkmf.apps.aro.v20261001.113i.j5059456.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-cfkmf.apps.aro.v20261001.113i.j5059456.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-cfkmf.apps.aro.v20261001.113i.j5059456.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-cfkmf.apps.aro.v20261001.113i.j5059456.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-cfkmf.apps.aro.v20261001.113i.j5059456.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x14a4f39220e0>{
            msg: "DNS for agnhost-e2e-sample-app-cfkmf.apps.aro.v20261001.113i.j5059456.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-cfkmf.apps.aro.v20261001.113i.j5059456.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x14a4f3922100>: 
    DNS for route host agnhost-e2e-sample-app-cfkmf.apps.aro.v20261001.113i.j5059456.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-cfkmf.apps.aro.v20261001.113i.j5059456.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-cfkmf.apps.aro.v20261001.113i.j5059456.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-cfkmf.apps.aro.v20261001.113i.j5059456.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-cfkmf.apps.aro.v20261001.113i.j5059456.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-cfkmf.apps.aro.v20261001.113i.j5059456.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x14a4f39220e0>{
            msg: "DNS for agnhost-e2e-sample-app-cfkmf.apps.aro.v20261001.113i.j5059456.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-cfkmf.apps.aro.v20261001.113i.j5059456.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x4eb350769e0>: 
    DNS for route host agnhost-e2e-sample-app-wbh7b.apps.aro.v20260901.v0z4.j6109568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-wbh7b.apps.aro.v20260901.v0z4.j6109568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wbh7b.apps.aro.v20260901.v0z4.j6109568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-wbh7b.apps.aro.v20260901.v0z4.j6109568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-wbh7b.apps.aro.v20260901.v0z4.j6109568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wbh7b.apps.aro.v20260901.v0z4.j6109568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x4eb350769c0>{
            msg: "DNS for agnhost-e2e-sample-app-wbh7b.apps.aro.v20260901.v0z4.j6109568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wbh7b.apps.aro.v20260901.v0z4.j6109568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0x4eb350769e0>: 
    DNS for route host agnhost-e2e-sample-app-wbh7b.apps.aro.v20260901.v0z4.j6109568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-wbh7b.apps.aro.v20260901.v0z4.j6109568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wbh7b.apps.aro.v20260901.v0z4.j6109568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-wbh7b.apps.aro.v20260901.v0z4.j6109568.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-wbh7b.apps.aro.v20260901.v0z4.j6109568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wbh7b.apps.aro.v20260901.v0z4.j6109568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x4eb350769c0>{
            msg: "DNS for agnhost-e2e-sample-app-wbh7b.apps.aro.v20260901.v0z4.j6109568.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wbh7b.apps.aro.v20260901.v0z4.j6109568.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
Affected runs (5)
Date (UTC)Associated PRProw job
PR #75192110371366967447552
PR #74302110316641895059456
PR #74922110275068176109568
PR #75152110253677582225408
PR #72482110168566228586496
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: IndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)
e2e54.20%5 of 119 job runs affectedIndeterminateSignal: Indeterminate — seen in 4 prior week(s); active 1 day(s)Show trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 5none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x3ad81c033ca0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-gsxwfjg2wp8q" to finish creating: GET https://rp.j6056960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b803943b-9e39-4dba-aaa8-6afe2e02cb47
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b803943b-9e39-4dba-aaa8-6afe2e02cb47",
      "name": "b803943b-9e39-4dba-aaa8-6afe2e02cb47",
      "status": "Failed",
      "startTime": "2026-10-14T22:01:54.508607559Z",
      "endTime": "2026-10-14T22:30:54.403314647Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u0w4f2e8s8w0a8w-np-delete-main-tbj7t-gshqg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-gsxwfjg2wp8q--managed/providers/Microsoft.Compute/virtualMachines/u0w4f2e8s8w0a8w-np-delete-main-tbj7t-gshqg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u0w4f2e8s8w0a8w-np-delete-main-tbj7t-gshqg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-gsxwfjg2wp8q--managed/providers/Microsoft.Compute/virtualMachines/u0w4f2e8s8w0a8w-np-delete-main-tbj7t-gshqg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6056960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b803943b-9e39-4dba-aaa8-6afe2e02cb47
            correlation ID: 988bdfac-593d-495e-9f75-b7aaa11925ed
            client request ID: 251c1e65-e123-4756-a69e-5f6923040f9c
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x3ad81c033ca0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-gsxwfjg2wp8q" to finish creating: GET https://rp.j6056960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b803943b-9e39-4dba-aaa8-6afe2e02cb47
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b803943b-9e39-4dba-aaa8-6afe2e02cb47",
      "name": "b803943b-9e39-4dba-aaa8-6afe2e02cb47",
      "status": "Failed",
      "startTime": "2026-10-14T22:01:54.508607559Z",
      "endTime": "2026-10-14T22:30:54.403314647Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine u0w4f2e8s8w0a8w-np-delete-main-tbj7t-gshqg: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-gsxwfjg2wp8q--managed/providers/Microsoft.Compute/virtualMachines/u0w4f2e8s8w0a8w-np-delete-main-tbj7t-gshqg to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine u0w4f2e8s8w0a8w-np-delete-main-tbj7t-gshqg: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-gsxwfjg2wp8q--managed/providers/Microsoft.Compute/virtualMachines/u0w4f2e8s8w0a8w-np-delete-main-tbj7t-gshqg to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6056960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b803943b-9e39-4dba-aaa8-6afe2e02cb47
            correlation ID: 988bdfac-593d-495e-9f75-b7aaa11925ed
            client request ID: 251c1e65-e123-4756-a69e-5f6923040f9c
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x235e937976c0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-68hx6zjhrzrm" to finish creating: GET https://rp.j6618368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/14b98349-fe00-4d10-b2ba-bdfa836e3ef6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/14b98349-fe00-4d10-b2ba-bdfa836e3ef6",
      "name": "14b98349-fe00-4d10-b2ba-bdfa836e3ef6",
      "status": "Failed",
      "startTime": "2026-10-14T19:36:58.429408765Z",
      "endTime": "2026-10-14T20:05:59.913923951Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n6f3b4y4h9b0a8m-np-deletable-v7kcg-vdl7x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-68hx6zjhrzrm--managed/providers/Microsoft.Compute/virtualMachines/n6f3b4y4h9b0a8m-np-deletable-v7kcg-vdl7x to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n6f3b4y4h9b0a8m-np-deletable-v7kcg-vdl7x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-68hx6zjhrzrm--managed/providers/Microsoft.Compute/virtualMachines/n6f3b4y4h9b0a8m-np-deletable-v7kcg-vdl7x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6618368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/14b98349-fe00-4d10-b2ba-bdfa836e3ef6
            correlation ID: b0044d15-3081-45ef-b76a-eef2cf552d58
            client request ID: 9699734b-7b5f-4b49-8b9d-898444b64f0d
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x235e937976c0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-68hx6zjhrzrm" to finish creating: GET https://rp.j6618368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/14b98349-fe00-4d10-b2ba-bdfa836e3ef6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/14b98349-fe00-4d10-b2ba-bdfa836e3ef6",
      "name": "14b98349-fe00-4d10-b2ba-bdfa836e3ef6",
      "status": "Failed",
      "startTime": "2026-10-14T19:36:58.429408765Z",
      "endTime": "2026-10-14T20:05:59.913923951Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine n6f3b4y4h9b0a8m-np-deletable-v7kcg-vdl7x: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-68hx6zjhrzrm--managed/providers/Microsoft.Compute/virtualMachines/n6f3b4y4h9b0a8m-np-deletable-v7kcg-vdl7x to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine n6f3b4y4h9b0a8m-np-deletable-v7kcg-vdl7x: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-68hx6zjhrzrm--managed/providers/Microsoft.Compute/virtualMachines/n6f3b4y4h9b0a8m-np-deletable-v7kcg-vdl7x to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6618368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/14b98349-fe00-4d10-b2ba-bdfa836e3ef6
            correlation ID: b0044d15-3081-45ef-b76a-eef2cf552d58
            client request ID: 9699734b-7b5f-4b49-8b9d-898444b64f0d
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x2a84e4eaf300>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-kmw59x46gl29" to finish creating: GET https://rp.j6626944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c0d9fdd7-ab89-43fd-bb14-1369b1b457a6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c0d9fdd7-ab89-43fd-bb14-1369b1b457a6",
      "name": "c0d9fdd7-ab89-43fd-bb14-1369b1b457a6",
      "status": "Failed",
      "startTime": "2026-10-14T12:52:44.69049902Z",
      "endTime": "2026-10-14T13:21:50.180181739Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine t2p8x5c0u4g8u4j-np-delete-main-tbrj2-hkw4h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-kmw59x46gl29--managed/providers/Microsoft.Compute/virtualMachines/t2p8x5c0u4g8u4j-np-delete-main-tbrj2-hkw4h to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine t2p8x5c0u4g8u4j-np-delete-main-tbrj2-hkw4h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-kmw59x46gl29--managed/providers/Microsoft.Compute/virtualMachines/t2p8x5c0u4g8u4j-np-delete-main-tbrj2-hkw4h to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6626944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c0d9fdd7-ab89-43fd-bb14-1369b1b457a6
            correlation ID: df4065fe-b20a-424e-a40c-325e8f340d1d
            client request ID: 8a51c2cf-89b8-4cc4-96a9-004376aa27f0
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x2a84e4eaf300>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-kmw59x46gl29" to finish creating: GET https://rp.j6626944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c0d9fdd7-ab89-43fd-bb14-1369b1b457a6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c0d9fdd7-ab89-43fd-bb14-1369b1b457a6",
      "name": "c0d9fdd7-ab89-43fd-bb14-1369b1b457a6",
      "status": "Failed",
      "startTime": "2026-10-14T12:52:44.69049902Z",
      "endTime": "2026-10-14T13:21:50.180181739Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine t2p8x5c0u4g8u4j-np-delete-main-tbrj2-hkw4h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-kmw59x46gl29--managed/providers/Microsoft.Compute/virtualMachines/t2p8x5c0u4g8u4j-np-delete-main-tbrj2-hkw4h to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine t2p8x5c0u4g8u4j-np-delete-main-tbrj2-hkw4h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-kmw59x46gl29--managed/providers/Microsoft.Compute/virtualMachines/t2p8x5c0u4g8u4j-np-delete-main-tbrj2-hkw4h to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6626944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c0d9fdd7-ab89-43fd-bb14-1369b1b457a6
            correlation ID: df4065fe-b20a-424e-a40c-325e8f340d1d
            client request ID: 8a51c2cf-89b8-4cc4-96a9-004376aa27f0
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool4
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges1
Affected runs (5)
Date (UTC)Associated PRProw job
PR #75382110474243606056960
PR #74772110436010226618368
PR #75252110338041116626944
PR #74972110305876723109888
PR #72482110168566228586496
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)
e2e43.36%4 of 119 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)Show trend detailsOct 8: 2 · Oct 9: 1 · Oct 10: 1 · Oct 11: 1 · Oct 12: 1 · Oct 13: 1 · Oct 14: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0x5bcdb5a5440>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-5gxqdd8p2qbn" to finish creating: GET https://rp.j3055232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9da300d2-61ca-49b6-8cec-c2ecaffe701e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9da300d2-61ca-49b6-8cec-c2ecaffe701e",
      "name": "9da300d2-61ca-49b6-8cec-c2ecaffe701e",
      "status": "Failed",
      "startTime": "2026-10-14T21:26:38.348499296Z",
      "endTime": "2026-10-14T21:55:39.3330542Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3055232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9da300d2-61ca-49b6-8cec-c2ecaffe701e
            correlation ID: 7d23b8d5-7dba-484d-af3a-9becf223a3c1
            client request ID: 66e7e251-c1da-4383-aef0-9b31e92742c7
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0x5bcdb5a5440>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-5gxqdd8p2qbn" to finish creating: GET https://rp.j3055232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9da300d2-61ca-49b6-8cec-c2ecaffe701e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9da300d2-61ca-49b6-8cec-c2ecaffe701e",
      "name": "9da300d2-61ca-49b6-8cec-c2ecaffe701e",
      "status": "Failed",
      "startTime": "2026-10-14T21:26:38.348499296Z",
      "endTime": "2026-10-14T21:55:39.3330542Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3055232.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9da300d2-61ca-49b6-8cec-c2ecaffe701e
            correlation ID: 7d23b8d5-7dba-484d-af3a-9becf223a3c1
            client request ID: 66e7e251-c1da-4383-aef0-9b31e92742c7
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-vtxdl7" for cluster "cluster-candidate-5-0-vtxdl7"
Unexpected error:
    <*fmt.wrapError | 0x23d38cc9d0c0>: 
    failed to create NodePool np-vtxdl7: failed waiting for nodepool="np-vtxdl7" for cluster "cluster-candidate-5-0-vtxdl7" in resourcegroup="rg-candidate-5-0-llmp2gg5r5wc" to finish creating: GET https://rp.j0018432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/53b3d6e3-5baf-4a04-b762-6c77f3e9c4a2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/53b3d6e3-5baf-4a04-b762-6c77f3e9c4a2",
      "name": "53b3d6e3-5baf-4a04-b762-6c77f3e9c4a2",
      "status": "Failed",
      "startTime": "2026-10-14T15:40:21.526109364Z",
      "endTime": "2026-10-14T16:09:23.214850022Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0018432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/53b3d6e3-5baf-4a04-b762-6c77f3e9c4a2
            correlation ID: 1427370e-268a-4518-bc82-b69439593955
            client request ID: 074febab-8e11-4f6f-b6d6-be4d97f6fe52
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-vtxdl7" for cluster "cluster-candidate-5-0-vtxdl7"
Unexpected error:
    <*fmt.wrapError | 0x23d38cc9d0c0>: 
    failed to create NodePool np-vtxdl7: failed waiting for nodepool="np-vtxdl7" for cluster "cluster-candidate-5-0-vtxdl7" in resourcegroup="rg-candidate-5-0-llmp2gg5r5wc" to finish creating: GET https://rp.j0018432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/53b3d6e3-5baf-4a04-b762-6c77f3e9c4a2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/53b3d6e3-5baf-4a04-b762-6c77f3e9c4a2",
      "name": "53b3d6e3-5baf-4a04-b762-6c77f3e9c4a2",
      "status": "Failed",
      "startTime": "2026-10-14T15:40:21.526109364Z",
      "endTime": "2026-10-14T16:09:23.214850022Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0018432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/53b3d6e3-5baf-4a04-b762-6c77f3e9c4a2
            correlation ID: 1427370e-268a-4518-bc82-b69439593955
            client request ID: 074febab-8e11-4f6f-b6d6-be4d97f6fe52
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xf6a2f24c800>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-lpcs9lxsfkb6" to finish creating: GET https://rp.j5913088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7476d679-0544-4ea6-98bc-8369b57375b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7476d679-0544-4ea6-98bc-8369b57375b6",
      "name": "7476d679-0544-4ea6-98bc-8369b57375b6",
      "status": "Failed",
      "startTime": "2026-10-14T14:43:09.62497089Z",
      "endTime": "2026-10-14T15:12:14.517634212Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5913088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7476d679-0544-4ea6-98bc-8369b57375b6
            correlation ID: b2aadf48-cb5e-49df-828e-ed7c131078ce
            client request ID: a92178cd-8f13-4632-bde1-e16a3dc65cc2
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xf6a2f24c800>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-lpcs9lxsfkb6" to finish creating: GET https://rp.j5913088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7476d679-0544-4ea6-98bc-8369b57375b6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7476d679-0544-4ea6-98bc-8369b57375b6",
      "name": "7476d679-0544-4ea6-98bc-8369b57375b6",
      "status": "Failed",
      "startTime": "2026-10-14T14:43:09.62497089Z",
      "endTime": "2026-10-14T15:12:14.517634212Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j5913088.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7476d679-0544-4ea6-98bc-8369b57375b6
            correlation ID: b2aadf48-cb5e-49df-828e-ed7c131078ce
            client request ID: a92178cd-8f13-4632-bde1-e16a3dc65cc2
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (4)
Date (UTC)Associated PRProw job
PR #68662110463708953055232
PR #75292110377553440018432
PR #67462110361307805913088
PR #75192110334779541950464
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
e2e21.68%2 of 119 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 42 · Oct 13: 0 · Oct 14: 9none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:114]: failed to create cluster "sre-version-pin-vc9jq7" at 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x2da62fe74520>: 
    failed to create HCP cluster sre-version-pin-vc9jq7: failed waiting for cluster="sre-version-pin-vc9jq7" in resourcegroup="admin-api-version-pin-ncqjrnsn47gn" to finish creating: GET https://rp.j6618368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3a2cf661-5481-4ce5-9c2c-34f8ee3716ac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3a2cf661-5481-4ce5-9c2c-34f8ee3716ac",
      "name": "3a2cf661-5481-4ce5-9c2c-34f8ee3716ac",
      "status": "Failed",
      "startTime": "2026-10-14T19:11:51.389728795Z",
      "endTime": "2026-10-14T19:41:00.041259206Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6618368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3a2cf661-5481-4ce5-9c2c-34f8ee3716ac
            correlation ID: 3bd031da-51c1-4750-a9ad-a1b50d360688
            client request ID: c18e0ef2-a2d7-4b12-a856-0335c427f46a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:114]: failed to create cluster "sre-version-pin-vc9jq7" at 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x2da62fe74520>: 
    failed to create HCP cluster sre-version-pin-vc9jq7: failed waiting for cluster="sre-version-pin-vc9jq7" in resourcegroup="admin-api-version-pin-ncqjrnsn47gn" to finish creating: GET https://rp.j6618368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3a2cf661-5481-4ce5-9c2c-34f8ee3716ac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3a2cf661-5481-4ce5-9c2c-34f8ee3716ac",
      "name": "3a2cf661-5481-4ce5-9c2c-34f8ee3716ac",
      "status": "Failed",
      "startTime": "2026-10-14T19:11:51.389728795Z",
      "endTime": "2026-10-14T19:41:00.041259206Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6618368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3a2cf661-5481-4ce5-9c2c-34f8ee3716ac
            correlation ID: 3bd031da-51c1-4750-a9ad-a1b50d360688
            client request ID: c18e0ef2-a2d7-4b12-a856-0335c427f46a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:152]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x3dad90af33a0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-sbj86zcf4vq4" to finish creating: GET https://rp.j6618368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/18cf34f6-b63a-4010-8930-865b2da0c38d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/18cf34f6-b63a-4010-8930-865b2da0c38d",
      "name": "18cf34f6-b63a-4010-8930-865b2da0c38d",
      "status": "Failed",
      "startTime": "2026-10-14T19:09:44.204708518Z",
      "endTime": "2026-10-14T19:38:50.032848393Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6618368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/18cf34f6-b63a-4010-8930-865b2da0c38d
            correlation ID: 8581e472-de79-4013-9c83-cf00277e2806
            client request ID: 3009aa8f-ca8e-40df-af26-d6a54c07c012
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:152]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x3dad90af33a0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-sbj86zcf4vq4" to finish creating: GET https://rp.j6618368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/18cf34f6-b63a-4010-8930-865b2da0c38d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/18cf34f6-b63a-4010-8930-865b2da0c38d",
      "name": "18cf34f6-b63a-4010-8930-865b2da0c38d",
      "status": "Failed",
      "startTime": "2026-10-14T19:09:44.204708518Z",
      "endTime": "2026-10-14T19:38:50.032848393Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6618368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/18cf34f6-b63a-4010-8930-865b2da0c38d
            correlation ID: 8581e472-de79-4013-9c83-cf00277e2806
            client request ID: 3009aa8f-ca8e-40df-af26-d6a54c07c012
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-mxm2nz" with version 4.21.36 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x2e81b57e6100>: 
    failed to create HCP cluster cluster-zstream-4-21-mxm2nz: failed waiting for cluster="cluster-zstream-4-21-mxm2nz" in resourcegroup="rg-zstream-upgrade-4-21-9f22s7ghtq4x" to finish creating: GET https://rp.j6618368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e22f032f-fd1f-4f7a-a0f0-79ca0d681ff5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e22f032f-fd1f-4f7a-a0f0-79ca0d681ff5",
      "name": "e22f032f-fd1f-4f7a-a0f0-79ca0d681ff5",
      "status": "Failed",
      "startTime": "2026-10-14T19:09:23.643995805Z",
      "endTime": "2026-10-14T19:38:30.41816485Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6618368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e22f032f-fd1f-4f7a-a0f0-79ca0d681ff5
            correlation ID: 40e10400-2a72-4df0-a76e-c12b12d5889c
            client request ID: 5416d7ee-a6cc-4125-9004-a1fe5836d05b
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-mxm2nz" with version 4.21.36 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x2e81b57e6100>: 
    failed to create HCP cluster cluster-zstream-4-21-mxm2nz: failed waiting for cluster="cluster-zstream-4-21-mxm2nz" in resourcegroup="rg-zstream-upgrade-4-21-9f22s7ghtq4x" to finish creating: GET https://rp.j6618368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e22f032f-fd1f-4f7a-a0f0-79ca0d681ff5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e22f032f-fd1f-4f7a-a0f0-79ca0d681ff5",
      "name": "e22f032f-fd1f-4f7a-a0f0-79ca0d681ff5",
      "status": "Failed",
      "startTime": "2026-10-14T19:09:23.643995805Z",
      "endTime": "2026-10-14T19:38:30.41816485Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster has no control plane endpoint host"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6618368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e22f032f-fd1f-4f7a-a0f0-79ca0d681ff5
            correlation ID: 40e10400-2a72-4df0-a76e-c12b12d5889c
            client request ID: 5416d7ee-a6cc-4125-9004-a1fe5836d05b
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 4.212
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.202
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.212
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE starts a control plane z-stream upgrade with an SRE pin1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74772110436010226618368
PR #74772110340691224694784
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e21.68%2 of 119 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0x15cfa7358820>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-kkjsbcrkrfqs" to finish creating: GET https://rp.j3670144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d48f5df-4419-42d0-908d-f55c9e58b43d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d48f5df-4419-42d0-908d-f55c9e58b43d",
      "name": "3d48f5df-4419-42d0-908d-f55c9e58b43d",
      "status": "Failed",
      "startTime": "2026-10-14T20:00:20.459546459Z",
      "endTime": "2026-10-14T20:29:21.27372498Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.37 is Partial (want Completed), started 24m22s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3670144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d48f5df-4419-42d0-908d-f55c9e58b43d
            correlation ID: 7a9ff375-e54e-4cef-a85f-da37d27bb541
            client request ID: 20f4c917-baec-45d6-a1e1-7f0ef2326c1c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0x15cfa7358820>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-kkjsbcrkrfqs" to finish creating: GET https://rp.j3670144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d48f5df-4419-42d0-908d-f55c9e58b43d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d48f5df-4419-42d0-908d-f55c9e58b43d",
      "name": "3d48f5df-4419-42d0-908d-f55c9e58b43d",
      "status": "Failed",
      "startTime": "2026-10-14T20:00:20.459546459Z",
      "endTime": "2026-10-14T20:29:21.27372498Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.37 is Partial (want Completed), started 24m22s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3670144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d48f5df-4419-42d0-908d-f55c9e58b43d
            correlation ID: 7a9ff375-e54e-4cef-a85f-da37d27bb541
            client request ID: 20f4c917-baec-45d6-a1e1-7f0ef2326c1c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-8fzqsptw7bpr/cluster-candidate-5-0-g7bpwx should provision
Unexpected error:
    <*fmt.wrapError | 0x4abdbffea60>: 
    failed to create HCP cluster cluster-candidate-5-0-g7bpwx: failed waiting for cluster="cluster-candidate-5-0-g7bpwx" in resourcegroup="rg-candidate-5-0-8fzqsptw7bpr" to finish creating: GET https://rp.j3579520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b237387d-d51f-41fb-8762-715d9d2b7454
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b237387d-d51f-41fb-8762-715d9d2b7454",
      "name": "b237387d-d51f-41fb-8762-715d9d2b7454",
      "status": "Failed",
      "startTime": "2026-10-14T07:05:39.437908469Z",
      "endTime": "2026-10-14T07:34:48.061766511Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.6 is Partial (want Completed), started 12m48s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3579520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b237387d-d51f-41fb-8762-715d9d2b7454
            correlation ID: 92af7c84-e19d-482a-a1fe-a87a2b603856
            client request ID: 2dc5545b-5fc8-4471-80a2-89372581489c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-8fzqsptw7bpr/cluster-candidate-5-0-g7bpwx should provision
Unexpected error:
    <*fmt.wrapError | 0x4abdbffea60>: 
    failed to create HCP cluster cluster-candidate-5-0-g7bpwx: failed waiting for cluster="cluster-candidate-5-0-g7bpwx" in resourcegroup="rg-candidate-5-0-8fzqsptw7bpr" to finish creating: GET https://rp.j3579520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b237387d-d51f-41fb-8762-715d9d2b7454
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b237387d-d51f-41fb-8762-715d9d2b7454",
      "name": "b237387d-d51f-41fb-8762-715d9d2b7454",
      "status": "Failed",
      "startTime": "2026-10-14T07:05:39.437908469Z",
      "endTime": "2026-10-14T07:34:48.061766511Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.6 is Partial (want Completed), started 12m48s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j3579520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b237387d-d51f-41fb-8762-715d9d2b7454
            correlation ID: 92af7c84-e19d-482a-a1fe-a87a2b603856
            client request ID: 2dc5545b-5fc8-4471-80a2-89372581489c
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #75292110447856593670144
PR #75042110251822873579520
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history
e2e21.68%2 of 119 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior historyShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 2 · Oct 13: 3 · Oct 14: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-7tqpsp" for cluster "cluster-hs-pre-7tqpsp"
Unexpected error:
    <*fmt.wrapError | 0x9fbbba82620>: 
    failed to create NodePool np-7tqpsp: failed waiting for nodepool="np-7tqpsp" for cluster "cluster-hs-pre-7tqpsp" in resourcegroup="rg-hs-pre--jlgwrtjbwxnf" to finish creating: GET https://rp.j4313856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bae4fd0e-d65b-410a-a9d7-cb38c5e195e6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bae4fd0e-d65b-410a-a9d7-cb38c5e195e6",
      "name": "bae4fd0e-d65b-410a-a9d7-cb38c5e195e6",
      "status": "Failed",
      "startTime": "2026-10-14T09:53:58.678670385Z",
      "endTime": "2026-10-14T10:23:01.968138297Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a0b7j6n3g2s4x1b-np-7tqpsp-rqshb-xb9b7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--jlgwrtjbwxnf-7tqpsp--managed/providers/Microsoft.Compute/virtualMachines/a0b7j6n3g2s4x1b-np-7tqpsp-rqshb-xb9b7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a0b7j6n3g2s4x1b-np-7tqpsp-rqshb-xb9b7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--jlgwrtjbwxnf-7tqpsp--managed/providers/Microsoft.Compute/virtualMachines/a0b7j6n3g2s4x1b-np-7tqpsp-rqshb-xb9b7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4313856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bae4fd0e-d65b-410a-a9d7-cb38c5e195e6
            correlation ID: fb25957f-8e08-4f05-8265-7dd58e255cf0
            client request ID: 4be80cba-579f-4cb7-8b46-0f96bfb70faa
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-7tqpsp" for cluster "cluster-hs-pre-7tqpsp"
Unexpected error:
    <*fmt.wrapError | 0x9fbbba82620>: 
    failed to create NodePool np-7tqpsp: failed waiting for nodepool="np-7tqpsp" for cluster "cluster-hs-pre-7tqpsp" in resourcegroup="rg-hs-pre--jlgwrtjbwxnf" to finish creating: GET https://rp.j4313856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bae4fd0e-d65b-410a-a9d7-cb38c5e195e6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bae4fd0e-d65b-410a-a9d7-cb38c5e195e6",
      "name": "bae4fd0e-d65b-410a-a9d7-cb38c5e195e6",
      "status": "Failed",
      "startTime": "2026-10-14T09:53:58.678670385Z",
      "endTime": "2026-10-14T10:23:01.968138297Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a0b7j6n3g2s4x1b-np-7tqpsp-rqshb-xb9b7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--jlgwrtjbwxnf-7tqpsp--managed/providers/Microsoft.Compute/virtualMachines/a0b7j6n3g2s4x1b-np-7tqpsp-rqshb-xb9b7 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a0b7j6n3g2s4x1b-np-7tqpsp-rqshb-xb9b7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--jlgwrtjbwxnf-7tqpsp--managed/providers/Microsoft.Compute/virtualMachines/a0b7j6n3g2s4x1b-np-7tqpsp-rqshb-xb9b7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j4313856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bae4fd0e-d65b-410a-a9d7-cb38c5e195e6
            correlation ID: fb25957f-8e08-4f05-8265-7dd58e255cf0
            client request ID: 4be80cba-579f-4cb7-8b46-0f96bfb70faa
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-cxf7bl" for cluster "cluster-hs-pre-cxf7bl"
Unexpected error:
    <*fmt.wrapError | 0x7e11cac4ba0>: 
    failed to create NodePool np-cxf7bl: failed waiting for nodepool="np-cxf7bl" for cluster "cluster-hs-pre-cxf7bl" in resourcegroup="rg-hs-pre--zccxz29md4tk" to finish creating: GET https://rp.j1740672.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a6b9db2-bba7-47f4-b565-c775480b382a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a6b9db2-bba7-47f4-b565-c775480b382a",
      "name": "2a6b9db2-bba7-47f4-b565-c775480b382a",
      "status": "Failed",
      "startTime": "2026-10-14T08:08:22.741490309Z",
      "endTime": "2026-10-14T08:37:29.934217142Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d9f9j4p4h9j5g5y-np-cxf7bl-d29hq-jxkj6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--zccxz29md4tk-cxf7bl--managed/providers/Microsoft.Compute/virtualMachines/d9f9j4p4h9j5g5y-np-cxf7bl-d29hq-jxkj6 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d9f9j4p4h9j5g5y-np-cxf7bl-d29hq-jxkj6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--zccxz29md4tk-cxf7bl--managed/providers/Microsoft.Compute/virtualMachines/d9f9j4p4h9j5g5y-np-cxf7bl-d29hq-jxkj6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1740672.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a6b9db2-bba7-47f4-b565-c775480b382a
            correlation ID: ca4a0e92-cce5-474f-a388-05ef9eae8408
            client request ID: 843e4749-4432-4dd5-a384-22418cffc290
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-cxf7bl" for cluster "cluster-hs-pre-cxf7bl"
Unexpected error:
    <*fmt.wrapError | 0x7e11cac4ba0>: 
    failed to create NodePool np-cxf7bl: failed waiting for nodepool="np-cxf7bl" for cluster "cluster-hs-pre-cxf7bl" in resourcegroup="rg-hs-pre--zccxz29md4tk" to finish creating: GET https://rp.j1740672.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a6b9db2-bba7-47f4-b565-c775480b382a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a6b9db2-bba7-47f4-b565-c775480b382a",
      "name": "2a6b9db2-bba7-47f4-b565-c775480b382a",
      "status": "Failed",
      "startTime": "2026-10-14T08:08:22.741490309Z",
      "endTime": "2026-10-14T08:37:29.934217142Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d9f9j4p4h9j5g5y-np-cxf7bl-d29hq-jxkj6: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--zccxz29md4tk-cxf7bl--managed/providers/Microsoft.Compute/virtualMachines/d9f9j4p4h9j5g5y-np-cxf7bl-d29hq-jxkj6 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d9f9j4p4h9j5g5y-np-cxf7bl-d29hq-jxkj6: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--zccxz29md4tk-cxf7bl--managed/providers/Microsoft.Compute/virtualMachines/d9f9j4p4h9j5g5y-np-cxf7bl-d29hq-jxkj6 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j1740672.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2a6b9db2-bba7-47f4-b565-c775480b382a
            correlation ID: ca4a0e92-cce5-474f-a388-05ef9eae8408
            client request ID: 843e4749-4432-4dd5-a384-22418cffc290
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #75152110290387544313856
PR #74512110267140211740672
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod networks, namespaces; [descendantResourc...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
e2e21.68%2 of 119 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsOct 8: 1 · Oct 9: 1 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 1 · Oct 14: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:320]: Unexpected error:
    <*errors.joinError | 0x1e1ed182150>: 
    resource group "rg-candidate-4-21-xnx2svpmndkd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-21-bqkz5m" in resourcegroup="rg-candidate-4-21-xnx2svpmndkd" to finish deleting: GET https://rp.j6397952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/aefc9a01-d084-42f4-9d9d-673831774ba7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/aefc9a01-d084-42f4-9d9d-673831774ba7",
      "name": "aefc9a01-d084-42f4-9d9d-673831774ba7",
      "status": "Failed",
      "startTime": "2026-10-14T20:07:21.117969363Z",
      "endTime": "2026-10-14T20:31:23.904548294Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6397952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/aefc9a01-d084-42f4-9d9d-673831774ba7
            correlation ID: e7f218c6-4d7a-4ce1-8044-c7ec05ec7356
            client request ID: e554513d-44da-4c58-8dae-c72863b0374d
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:320]: Unexpected error:
    <*errors.joinError | 0x1e1ed182150>: 
    resource group "rg-candidate-4-21-xnx2svpmndkd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-21-bqkz5m" in resourcegroup="rg-candidate-4-21-xnx2svpmndkd" to finish deleting: GET https://rp.j6397952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/aefc9a01-d084-42f4-9d9d-673831774ba7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/aefc9a01-d084-42f4-9d9d-673831774ba7",
      "name": "aefc9a01-d084-42f4-9d9d-673831774ba7",
      "status": "Failed",
      "startTime": "2026-10-14T20:07:21.117969363Z",
      "endTime": "2026-10-14T20:31:23.904548294Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j6397952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/aefc9a01-d084-42f4-9d9d-673831774ba7
            correlation ID: e7f218c6-4d7a-4ce1-8044-c7ec05ec7356
            client request ID: e554513d-44da-4c58-8dae-c72863b0374d
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
    <*errors.joinError | 0x50b30376378>: 
    resource group "rg-candidate-4-23-zgqgjh9b82nm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-v7mn4s" in resourcegroup="rg-candidate-4-23-zgqgjh9b82nm" to finish deleting: GET https://rp.j9234816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/90b34b3b-49e0-48b7-b175-fed3e7efcffe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/90b34b3b-49e0-48b7-b175-fed3e7efcffe",
      "name": "90b34b3b-49e0-48b7-b175-fed3e7efcffe",
      "status": "Failed",
      "startTime": "2026-10-14T15:55:23.629401759Z",
      "endTime": "2026-10-14T16:19:25.512560555Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9234816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/90b34b3b-49e0-48b7-b175-fed3e7efcffe
            correlation ID: 18b3d50f-1bf0-4dc5-95df-d87efa3be15d
            client request ID: eef778ec-0385-499a-a74c-a4ea9c9eb634
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
    <*errors.joinError | 0x50b30376378>: 
    resource group "rg-candidate-4-23-zgqgjh9b82nm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-v7mn4s" in resourcegroup="rg-candidate-4-23-zgqgjh9b82nm" to finish deleting: GET https://rp.j9234816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/90b34b3b-49e0-48b7-b175-fed3e7efcffe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/90b34b3b-49e0-48b7-b175-fed3e7efcffe",
      "name": "90b34b3b-49e0-48b7-b175-fed3e7efcffe",
      "status": "Failed",
      "startTime": "2026-10-14T15:55:23.629401759Z",
      "endTime": "2026-10-14T16:19:25.512560555Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j9234816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/90b34b3b-49e0-48b7-b175-fed3e7efcffe
            correlation ID: 18b3d50f-1bf0-4dc5-95df-d87efa3be15d
            client request ID: eef778ec-0385-499a-a74c-a4ea9c9eb634
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
Affected runs (2)
Date (UTC)Associated PRProw job
PR #66352110445968326397952
PR #74282110376503379234816
There is no failure as the matcher passed to Eventually succeeded on its most recent iteration
full failure pattern:
There is no failure as the matcher passed to Eventually succeeded on its most recent iteration
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e21.68%2 of 119 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:394]: Timed out after 90.003s.
delegation and authoritative NS answers should remain consistent for at least 30 seconds before issuance
There is no failure as the matcher passed to Eventually succeeded on its most recent iteration
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:394]: Timed out after 90.003s.
delegation and authoritative NS answers should remain consistent for at least 30 seconds before issuance
There is no failure as the matcher passed to Eventually succeeded on its most recent iteration
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:394]: Timed out after 90.000s.
delegation and authoritative NS answers should remain consistent for at least 30 seconds before issuance
There is no failure as the matcher passed to Eventually succeeded on its most recent iteration
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:394]: Timed out after 90.000s.
delegation and authoritative NS answers should remain consistent for at least 30 seconds before issuance
There is no failure as the matcher passed to Eventually succeeded on its most recent iteration
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #75442110520946510008320
PR #74922110389077940375552
ERROR CODE: DeploymentFailed; detail code HttpRequestException; detail message Error while processing network security perimeter association: An error occurred while sending the r...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code HttpRequestException; detail message Error while processing network security perimeter association: An error occurred while sending the request.; provider Microsoft.Network
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
provision21.68%2 of 119 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 2none
Full failure examples (2)
time=2026-10-14T19:14:18.297Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 description="Step nsp\n  Kind: ARM\n  Template: templates/mgmt-nsp.bicep\n  Parameters: configurations/mgmt-nsp.tmpl.bicepparam"
time=2026-10-14T19:14:18.788Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2
time=2026-10-14T19:14:19.241Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 deployment=cf8c9a34812ddee83e87f5e1fcb73992e7f0046ac9c8abee6011c27a19ac7c31 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9264896-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fcf8c9a34812ddee83e87f5e1fcb73992e7f0046ac9c8abee6011c27a19ac7c31
time=2026-10-14T19:14:49.370Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9264896-mgmt-2/providers/Microsoft.Resources/deployments/cf8c9a34812ddee83e87f5e1fcb73992e7f0046ac9c8abee6011c27a19ac7c31/operationStatuses/08584096016264886235\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9264896-mgmt-2/providers/Microsoft.Resources/deployments/profile-xpjnxfpmcvora\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9264896-mgmt-2/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j9264896-mgmt-2/resourceAssociations/nsp-j9264896-mgmt-2-ra-a9c395a5-2fe0-5927-97a4-6856e96269ae\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"HttpRequestException\\\",\\r\\n                \\\"message\\\": \\\"Error while processing request for association /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9264896-mgmt-2/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j9264896-mgmt-2/resourceAssociations/nsp-j9264896-mgmt-2-ra-a9c395a5-2fe0-5927-97a4-6856e96269ae, Error message: An error occurred while sending the request.\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
time=2026-10-14T15:57:00.478Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 description="Step nsp\n  Kind: ARM\n  Template: templates/mgmt-nsp.bicep\n  Parameters: configurations/mgmt-nsp.tmpl.bicepparam"
time=2026-10-14T15:57:01.032Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2
time=2026-10-14T15:57:01.677Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 deployment=9e882a024304e709a0349553a748b62ffe4602dc3aa507f98e24a10ee21a2af8 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j7217792-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9e882a024304e709a0349553a748b62ffe4602dc3aa507f98e24a10ee21a2af8
time=2026-10-14T15:57:32.150Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=nsp stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7217792-mgmt-2/providers/Microsoft.Resources/deployments/9e882a024304e709a0349553a748b62ffe4602dc3aa507f98e24a10ee21a2af8/operationStatuses/08584096134641768649\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7217792-mgmt-2/providers/Microsoft.Resources/deployments/profile-7bczjqgoi25xw\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7217792-mgmt-2/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j7217792-mgmt-2/resourceAssociations/nsp-j7217792-mgmt-2-ra-e5d09873-b78f-5e95-bab4-2921347dc438\\\",\\r\\n            \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n            \\\"details\\\": [\\r\\n              {\\r\\n                \\\"code\\\": \\\"HttpRequestException\\\",\\r\\n                \\\"message\\\": \\\"Error while processing request for association /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j7217792-mgmt-2/providers/Microsoft.Network/networkSecurityPerimeters/nsp-j7217792-mgmt-2/resourceAssociations/nsp-j7217792-mgmt-2-ra-e5d09873-b78f-5e95-bab4-2921347dc438, Error message: An error occurred while sending the request.\\\"\\r\\n              }\\r\\n            ]\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/nsp (stamp=2)2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #67462110439188909264896
PR #63642110390267277217792
ERROR CODE: DeadlineExceeded; detail message cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation "hypershift.openshi...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation "hypershift.openshift.io/cluster-size-override" is "e2e_minimal; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e10.84%1 of 119 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 5none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-8fmvrw" to 4.22
Unexpected error:
    <*fmt.wrapError | 0x189d56536760>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-22-8fmvrw" in resourcegroup="rg-cp-ystream-upgrade-4-22-m4rkshgrgs9x" to finish updating: GET https://rp.j8816896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/008de110-0fc7-4cfb-b1a8-16dd774cc912
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/008de110-0fc7-4cfb-b1a8-16dd774cc912",
      "name": "008de110-0fc7-4cfb-b1a8-16dd774cc912",
      "status": "Failed",
      "startTime": "2026-10-14T03:16:19.986510176Z",
      "endTime": "2026-10-14T04:00:20.242906036Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8816896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/008de110-0fc7-4cfb-b1a8-16dd774cc912
            correlation ID: edcafa24-51c5-48f1-9ce5-cdc56c9cfe21
            client request ID: d7ffd994-e6d8-430b-b52c-c25abf00d2d4
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-8fmvrw" to 4.22
Unexpected error:
    <*fmt.wrapError | 0x189d56536760>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-22-8fmvrw" in resourcegroup="rg-cp-ystream-upgrade-4-22-m4rkshgrgs9x" to finish updating: GET https://rp.j8816896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/008de110-0fc7-4cfb-b1a8-16dd774cc912
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/008de110-0fc7-4cfb-b1a8-16dd774cc912",
      "name": "008de110-0fc7-4cfb-b1a8-16dd774cc912",
      "status": "Failed",
      "startTime": "2026-10-14T03:16:19.986510176Z",
      "endTime": "2026-10-14T04:00:20.242906036Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8816896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/008de110-0fc7-4cfb-b1a8-16dd774cc912
            correlation ID: edcafa24-51c5-48f1-9ce5-cdc56c9cfe21
            client request ID: d7ffd994-e6d8-430b-b52c-c25abf00d2d4
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-v9gdsd" to 5.0
Unexpected error:
    <*fmt.wrapError | 0x89c8a7b4520>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-5-0-v9gdsd" in resourcegroup="rg-cp-ystream-upgrade-5-0-lrtt9r887h59" to finish updating: GET https://rp.j8816896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/149ad663-a4d9-4ebd-9101-8c188eed65d0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/149ad663-a4d9-4ebd-9101-8c188eed65d0",
      "name": "149ad663-a4d9-4ebd-9101-8c188eed65d0",
      "status": "Failed",
      "startTime": "2026-10-14T03:29:27.04612928Z",
      "endTime": "2026-10-14T04:13:34.922970159Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8816896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/149ad663-a4d9-4ebd-9101-8c188eed65d0
            correlation ID: b89bc015-38c4-4b8f-8cf6-fd7d29e794e7
            client request ID: c975ef93-0023-4a12-b6b6-8dfd3c0b6abf
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-v9gdsd" to 5.0
Unexpected error:
    <*fmt.wrapError | 0x89c8a7b4520>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-5-0-v9gdsd" in resourcegroup="rg-cp-ystream-upgrade-5-0-lrtt9r887h59" to finish updating: GET https://rp.j8816896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/149ad663-a4d9-4ebd-9101-8c188eed65d0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/149ad663-a4d9-4ebd-9101-8c188eed65d0",
      "name": "149ad663-a4d9-4ebd-9101-8c188eed65d0",
      "status": "Failed",
      "startTime": "2026-10-14T03:29:27.04612928Z",
      "endTime": "2026-10-14T04:13:34.922970159Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8816896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/149ad663-a4d9-4ebd-9101-8c188eed65d0
            correlation ID: b89bc015-38c4-4b8f-8cf6-fd7d29e794e7
            client request ID: c975ef93-0023-4a12-b6b6-8dfd3c0b6abf
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-trt4r8" to 4.23
Unexpected error:
    <*fmt.wrapError | 0xbc2497a0a40>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-trt4r8" in resourcegroup="rg-cp-ystream-upgrade-4-23-nm455m5dz4rg" to finish updating: GET https://rp.j8816896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ab2c099b-e08d-48bf-8615-536073622db3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ab2c099b-e08d-48bf-8615-536073622db3",
      "name": "ab2c099b-e08d-48bf-8615-536073622db3",
      "status": "Failed",
      "startTime": "2026-10-14T03:17:19.784745543Z",
      "endTime": "2026-10-14T04:01:20.236478176Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8816896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ab2c099b-e08d-48bf-8615-536073622db3
            correlation ID: b6e77c38-1227-46e6-9bc5-a4baba263145
            client request ID: fcf783cc-37c0-4ef5-8f68-49e5b907b57c
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:169]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-trt4r8" to 4.23
Unexpected error:
    <*fmt.wrapError | 0xbc2497a0a40>: 
    failed waiting for hcpCluster="cp-ystream-upgrade-4-23-trt4r8" in resourcegroup="rg-cp-ystream-upgrade-4-23-nm455m5dz4rg" to finish updating: GET https://rp.j8816896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ab2c099b-e08d-48bf-8615-536073622db3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ab2c099b-e08d-48bf-8615-536073622db3",
      "name": "ab2c099b-e08d-48bf-8615-536073622db3",
      "status": "Failed",
      "startTime": "2026-10-14T03:17:19.784745543Z",
      "endTime": "2026-10-14T04:01:20.236478176Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster update did not complete before the deadline; [hypershiftHostedCluster] hypershift HostedCluster annotation \"hypershift.openshift.io/cluster-size-override\" is \"e2e_minimal\", want unset"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j8816896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ab2c099b-e08d-48bf-8615-536073622db3
            correlation ID: b6e77c38-1227-46e6-9bc5-a4baba263145
            client request ID: fcf783cc-37c0-4ef5-8f68-49e5b907b57c
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69682110191032388816896
ERROR CODE: 429; provider Microsoft.ManagedIdentity
full failure pattern:
ERROR CODE: 429; provider Microsoft.ManagedIdentity
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
e2e10.84%1 of 119 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 1 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:119]: failed to get cluster-api-azure managed identity
Unexpected error:
    <*exported.ResponseError | 0x18d22f431050>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-56/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: 429
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "429",
        "message": "Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-56/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
            correlation ID: d277ac90-384a-44a9-9799-7698621627f9
            client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:119]: failed to get cluster-api-azure managed identity
Unexpected error:
    <*exported.ResponseError | 0x18d22f431050>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-56/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
    --------------------------------------------------------------------------------
    RESPONSE 429: 429 Too Many Requests
    ERROR CODE: 429
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "429",
        "message": "Request is temporarily throttled because subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds."
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard1-00-56/providers/Microsoft.ManagedIdentity/userAssignedIdentities/cluster-api-azure
            correlation ID: d277ac90-384a-44a9-9799-7698621627f9
            client request ID: unavailable
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool OS Disk Encryption should create a nodepool with customer-managed disk encryption via DES1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74882110168597706838016
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-<id> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-<id> to exist; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e10.84%1 of 119 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 2 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x22a4be5cfc40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-gc2cf577p6fs" to finish creating: GET https://rp.j0375552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1dec4678-5a13-4201-9600-24dcb5a26534
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1dec4678-5a13-4201-9600-24dcb5a26534",
      "name": "1dec4678-5a13-4201-9600-24dcb5a26534",
      "status": "Failed",
      "startTime": "2026-10-14T16:17:52.259133125Z",
      "endTime": "2026-10-14T16:46:52.310381675Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-dnjw8-hs9xx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-gc2cf577p6fs--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-dnjw8-hs9xx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-dnjw8-hs9xx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-gc2cf577p6fs--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-dnjw8-hs9xx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0375552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1dec4678-5a13-4201-9600-24dcb5a26534
            correlation ID: b81493f7-b048-4704-b54a-5a69ef8ad3b5
            client request ID: 4dd74744-6a7e-48d1-9b1b-985d5238151a
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001 cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x22a4be5cfc40>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001-gc2cf577p6fs" to finish creating: GET https://rp.j0375552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1dec4678-5a13-4201-9600-24dcb5a26534
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1dec4678-5a13-4201-9600-24dcb5a26534",
      "name": "1dec4678-5a13-4201-9600-24dcb5a26534",
      "status": "Failed",
      "startTime": "2026-10-14T16:17:52.259133125Z",
      "endTime": "2026-10-14T16:46:52.310381675Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-dnjw8-hs9xx: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-gc2cf577p6fs--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-dnjw8-hs9xx to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-dnjw8-hs9xx: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001-gc2cf577p6fs--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-dnjw8-hs9xx to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j0375552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/1dec4678-5a13-4201-9600-24dcb5a26534
            correlation ID: b81493f7-b048-4704-b54a-5a69ef8ad3b5
            client request ID: 4dd74744-6a7e-48d1-9b1b-985d5238151a
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74922110389077940375552
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet complete...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e10.84%1 of 119 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 1 · Oct 13: 1 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0x175f66e5a600>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-8brvsr9whg7b" to finish creating: GET https://rp.j2105600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cba8c76e-06a0-4c2a-b3e0-d699a7abb497
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cba8c76e-06a0-4c2a-b3e0-d699a7abb497",
      "name": "cba8c76e-06a0-4c2a-b3e0-d699a7abb497",
      "status": "Failed",
      "startTime": "2026-10-14T13:47:00.778542361Z",
      "endTime": "2026-10-14T14:16:06.716990034Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.37 is Partial (want Completed), started 17m32s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2105600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cba8c76e-06a0-4c2a-b3e0-d699a7abb497
            correlation ID: 113b689f-35da-4e51-8207-8a80d51d19ea
            client request ID: d9df4b89-ae88-4991-9a25-651fec75b816
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0x175f66e5a600>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-8brvsr9whg7b" to finish creating: GET https://rp.j2105600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cba8c76e-06a0-4c2a-b3e0-d699a7abb497
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cba8c76e-06a0-4c2a-b3e0-d699a7abb497",
      "name": "cba8c76e-06a0-4c2a-b3e0-d699a7abb497",
      "status": "Failed",
      "startTime": "2026-10-14T13:47:00.778542361Z",
      "endTime": "2026-10-14T14:16:06.716990034Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.37 is Partial (want Completed), started 17m32s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
        Azure request IDs:
          GET https://rp.j2105600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cba8c76e-06a0-4c2a-b3e0-d699a7abb497
            correlation ID: 113b689f-35da-4e51-8207-8a80d51d19ea
            client request ID: d9df4b89-ae88-4991-9a25-651fec75b816
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75222110350863582105600
ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible tha...
full failure pattern:
ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url>
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)
provision10.84%1 of 119 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)Show trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 2 · Oct 14: 1none
Full failure examples (1)
time=2026-10-14T20:02:04.703Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 description="Step infra\n  Kind: ARM\n  Template: templates/mgmt-infra.bicep\n  Parameters: configurations/mgmt-infra.tmpl.bicepparam"
time=2026-10-14T20:02:05.214Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2
time=2026-10-14T20:02:05.894Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 deployment=c992d66eb9a38cb8bacf66c4633734326584f1d12bbb6c98da67de8819d3eb79 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0582272-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fc992d66eb9a38cb8bacf66c4633734326584f1d12bbb6c98da67de8819d3eb79
time=2026-10-14T20:03:06.110Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0582272-mgmt-2/providers/Microsoft.Resources/deployments/c992d66eb9a38cb8bacf66c4633734326584f1d12bbb6c98da67de8819d3eb79/operationStatuses/08584095987599895937\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0582272-mgmt-2/providers/Microsoft.Resources/deployments/mgmt-kv-2awbxe73l63vi\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"VaultAlreadyExists\\\",\\r\\n            \\\"message\\\": \\\"The vault name 'ah-ci01-mg-j0582272-2' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0582272-mgmt-2/providers/Microsoft.Resources/deployments/msi-kv-g2p3wnorghzau\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"VaultAlreadyExists\\\",\\r\\n            \\\"message\\\": \\\"The vault name 'ah-ci01-mi-j0582272-2' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      },\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0582272-mgmt-2/providers/Microsoft.Resources/deployments/ah-ci01-me-j0582272-2\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"VaultAlreadyExists\\\",\\r\\n            \\\"message\\\": \\\"The vault name 'ah-ci01-me-j0582272-2' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/infra (stamp=2)1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74922110452210310582272
All ClusterOperators should report Available=True, but these are not available: [console (Available=False)]
full failure pattern:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False)]
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e10.84%1 of 119 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 8: 0 · Oct 9: 0 · Oct 10: 0 · Oct 11: 0 · Oct 12: 0 · Oct 13: 0 · Oct 14: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:378]: Timed out after 639.547s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:377 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False)]
Expected
    <[]string | len:1, cap:1>: [
        "console (Available=False)",
    ]
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:378]: Timed out after 639.547s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:377 with:
All ClusterOperators should report Available=True, but these are not available: [console (Available=False)]
Expected
    <[]string | len:1, cap:1>: [
        "console (Available=False)",
    ]
to be empty
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #75292110412598238253056