CIHealth
<
Single Day: Oct 9▾
Last 7 Days Weekly: Oct 5 - Oct 11 Sprint: Sep 28 - Oct 11 Single Day: Oct 9
Custom time window
> Reset
View JSON API

Environment: DEV

Failure patterns: 43 · Failures matched: 162 · Total runs: 59 · Runs affected: 38

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "engineering-workers": (must be a valid DNS RFC 1035 label); provider Microsof...
full failure pattern:
ERROR CODE: MultipleErrorsOccurred; detail code InvalidRequestContent; detail message Invalid value: "engineering-workers": (must be a valid DNS RFC 1035 label); provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e23.39%2 of 59 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 2 · Oct 8: 2 · Oct 9: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x2ae2e5f963c0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-2c747kttkrf7": PUT https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2c747kttkrf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x2ae2e5f963c0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-2c747kttkrf7": PUT https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2c747kttkrf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x95c6dd09a20>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-zsv74hnjtfn5": PUT https://rp.j2592384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zsv74hnjtfn5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x95c6dd09a20>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-zsv74hnjtfn5": PUT https://rp.j2592384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zsv74hnjtfn5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #73972108513285430054912
PR #73972108452968792592384
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced; ...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced; provider Microsoft.RedHatOpenShift
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.69%1 of 59 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 66none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x1d7c60a347e0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-wh6shv999rvk" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d322d275-9941-45bf-a054-aa9d4b20f7cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d322d275-9941-45bf-a054-aa9d4b20f7cc",
      "name": "d322d275-9941-45bf-a054-aa9d4b20f7cc",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:10.15141187Z",
      "endTime": "2026-10-09T18:31:17.847497361Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x1d7c60a347e0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-wh6shv999rvk" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d322d275-9941-45bf-a054-aa9d4b20f7cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d322d275-9941-45bf-a054-aa9d4b20f7cc",
      "name": "d322d275-9941-45bf-a054-aa9d4b20f7cc",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:10.15141187Z",
      "endTime": "2026-10-09T18:31:17.847497361Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x3aa3a58fd700>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-r9mpb5rglfkx" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/501ef370-ff82-48d0-af1d-38c7e8e27d51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/501ef370-ff82-48d0-af1d-38c7e8e27d51",
      "name": "501ef370-ff82-48d0-af1d-38c7e8e27d51",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:09.983214688Z",
      "endTime": "2026-10-09T18:31:17.777246897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x3aa3a58fd700>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-r9mpb5rglfkx" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/501ef370-ff82-48d0-af1d-38c7e8e27d51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/501ef370-ff82-48d0-af1d-38c7e8e27d51",
      "name": "501ef370-ff82-48d0-af1d-38c7e8e27d51",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:09.983214688Z",
      "endTime": "2026-10-09T18:31:17.777246897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0x3f8cfca18160>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-d9vjk88cdxmv" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/715e9b71-ab21-4169-9e2e-f3f804cb80c0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/715e9b71-ab21-4169-9e2e-f3f804cb80c0",
      "name": "715e9b71-ab21-4169-9e2e-f3f804cb80c0",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:40.105550226Z",
      "endTime": "2026-10-09T18:31:47.946839601Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0x3f8cfca18160>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-d9vjk88cdxmv" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/715e9b71-ab21-4169-9e2e-f3f804cb80c0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/715e9b71-ab21-4169-9e2e-f3f804cb80c0",
      "name": "715e9b71-ab21-4169-9e2e-f3f804cb80c0",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:40.105550226Z",
      "endTime": "2026-10-09T18:31:47.946839601Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (66)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.11
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should serve the default ingress certificate and allow a customer ingress certificate for 5.11
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and custom node pool osDisk size1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster and manage pull secrets1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with Image Registry not present1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to lifecycle and confirm external auth on a cluster1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to rotate KMS key for a cluster with version >= 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to use workload identity via the cluster OIDC issuer URL1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster using v20260901preview API and verify cluster health1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to reuse subnets and NSGs between clusters1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should respect cluster-wide node limits with nodepool autoscaling1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.20 minor to 4.21 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.21 minor to 4.22 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.22 minor to 4.23 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.22 minor to 5.0 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 5.0 minor to 5.1 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering want metrics from HCP to be present in Azure Monitor Workspace1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to retrieve serial console logs for a VM1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.231
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74402108607415241936896
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRerror for disabled boot diagnostics should contain 409 status code
full failure pattern:
error for disabled boot diagnostics should contain 409 status code
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
e2e11.69%1 of 59 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:456]: error for disabled boot diagnostics should contain 409 status code
Expected
    <string>: failed to send request: Get "https://admin.j1851392.hcpsvc.osadev.cloud/admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-serialconsole-bootdiag-mx7j6swwr25v/providers/Microsoft.RedHatOpenshift/hcpOpenShiftClusters/sre-hcp-cluster-bootdiag/serialconsole?vmName=d3i6t7c9q4h0k8j-worker-wrlpp-npg85": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
to contain substring
    <string>: 409
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:456]: error for disabled boot diagnostics should contain 409 status code
Expected
    <string>: failed to send request: Get "https://admin.j1851392.hcpsvc.osadev.cloud/admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-serialconsole-bootdiag-mx7j6swwr25v/providers/Microsoft.RedHatOpenshift/hcpOpenShiftClusters/sre-hcp-cluster-bootdiag/serialconsole?vmName=d3i6t7c9q4h0k8j-worker-wrlpp-npg85": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
to contain substring
    <string>: 409
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74372108374328641851392
Show signal detailsLikely regression — post-good=0; only seen in DEV; only seen in one PRerror running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>...
full failure pattern:
error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j5137536-mgmt-2","clusterName":"ci01-j513753
Signal: RegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PR
provision11.69%1 of 59 job runs affectedRegressionSignal: Regression — post-good=0; only seen in DEV; only seen in one PRShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 1none
Full failure examples (1)
time=2026-10-09T14:05:11.912Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-09T14:35:11.381Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-09T14:05:11.922891129Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5137536-mgmt-2\",\"clusterName\":\"ci01-j5137536-mgmt-2\",\"region\":\"westus3\"}\n{\"time\":\"2026-10-09T14:05:14.313036396Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5137536-mgmt-2\",\"clusterName\":\"ci01-j5137536-mgmt-2\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-09T14:05:14.653688404Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5137536-mgmt-2\",\"clusterName\":\"ci01-j5137536-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2)1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #67462108552522435137536
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
e2e711.86%7 of 59 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsOct 3: 10 · Oct 4: 61 · Oct 5: 276 · Oct 6: 236 · Oct 7: 126 · Oct 8: 12 · Oct 9: 9STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0x3113509f94c0>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-g776ncnttgrt" to finish creating: GET https://rp.j3788032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77fe8352-5c4d-493e-9b0c-7d5a039edff5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77fe8352-5c4d-493e-9b0c-7d5a039edff5",
      "name": "77fe8352-5c4d-493e-9b0c-7d5a039edff5",
      "status": "Failed",
      "startTime": "2026-10-09T19:34:01.982763751Z",
      "endTime": "2026-10-09T20:03:10.160667249Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-5j5ks-29l5h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-29l5h to exist\nMachine ephemeral-disk-ephemeral-np-5j5ks-4jtss: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-4jtss to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-5j5ks-29l5h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-29l5h to exist\nMachine ephemeral-disk-ephemeral-np-5j5ks-4jtss: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-4jtss to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0x3113509f94c0>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-g776ncnttgrt" to finish creating: GET https://rp.j3788032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77fe8352-5c4d-493e-9b0c-7d5a039edff5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77fe8352-5c4d-493e-9b0c-7d5a039edff5",
      "name": "77fe8352-5c4d-493e-9b0c-7d5a039edff5",
      "status": "Failed",
      "startTime": "2026-10-09T19:34:01.982763751Z",
      "endTime": "2026-10-09T20:03:10.160667249Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-5j5ks-29l5h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-29l5h to exist\nMachine ephemeral-disk-ephemeral-np-5j5ks-4jtss: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-4jtss to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-5j5ks-29l5h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-29l5h to exist\nMachine ephemeral-disk-ephemeral-np-5j5ks-4jtss: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-4jtss to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0xfc204541c00>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-j844k7kwk994" to finish creating: GET https://rp.j1306368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47be8315-7c7b-4770-bede-fbd3e338f83d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47be8315-7c7b-4770-bede-fbd3e338f83d",
      "name": "47be8315-7c7b-4770-bede-fbd3e338f83d",
      "status": "Failed",
      "startTime": "2026-10-09T14:07:16.315050699Z",
      "endTime": "2026-10-09T14:36:17.115068925Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c8a6f8l6z1i8v1z-cordon-np-szltg-ptx4m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-j844k7kwk994--managed/providers/Microsoft.Compute/virtualMachines/c8a6f8l6z1i8v1z-cordon-np-szltg-ptx4m to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c8a6f8l6z1i8v1z-cordon-np-szltg-ptx4m: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-j844k7kwk994--managed/providers/Microsoft.Compute/virtualMachines/c8a6f8l6z1i8v1z-cordon-np-szltg-ptx4m to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0xfc204541c00>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-j844k7kwk994" to finish creating: GET https://rp.j1306368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47be8315-7c7b-4770-bede-fbd3e338f83d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47be8315-7c7b-4770-bede-fbd3e338f83d",
      "name": "47be8315-7c7b-4770-bede-fbd3e338f83d",
      "status": "Failed",
      "startTime": "2026-10-09T14:07:16.315050699Z",
      "endTime": "2026-10-09T14:36:17.115068925Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c8a6f8l6z1i8v1z-cordon-np-szltg-ptx4m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-j844k7kwk994--managed/providers/Microsoft.Compute/virtualMachines/c8a6f8l6z1i8v1z-cordon-np-szltg-ptx4m to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c8a6f8l6z1i8v1z-cordon-np-szltg-ptx4m: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-j844k7kwk994--managed/providers/Microsoft.Compute/virtualMachines/c8a6f8l6z1i8v1z-cordon-np-szltg-ptx4m to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x58c914fe040>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-4zl2jn5xrcmj" to finish creating: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4fdf8129-ec4e-4f54-a67c-3179e8542d77
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4fdf8129-ec4e-4f54-a67c-3179e8542d77",
      "name": "4fdf8129-ec4e-4f54-a67c-3179e8542d77",
      "status": "Failed",
      "startTime": "2026-10-09T12:01:50.770272194Z",
      "endTime": "2026-10-09T12:30:51.906223646Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-qf6rt-prsfz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-4zl2jn5xrcmj--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-qf6rt-prsfz to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-qf6rt-prsfz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-4zl2jn5xrcmj--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-qf6rt-prsfz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x58c914fe040>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-4zl2jn5xrcmj" to finish creating: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4fdf8129-ec4e-4f54-a67c-3179e8542d77
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4fdf8129-ec4e-4f54-a67c-3179e8542d77",
      "name": "4fdf8129-ec4e-4f54-a67c-3179e8542d77",
      "status": "Failed",
      "startTime": "2026-10-09T12:01:50.770272194Z",
      "endTime": "2026-10-09T12:30:51.906223646Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-qf6rt-prsfz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-4zl2jn5xrcmj--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-qf6rt-prsfz to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-qf6rt-prsfz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-4zl2jn5xrcmj--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-qf6rt-prsfz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (9)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should create a cluster with private ingress using v20260901preview and verify the ingress is internal1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade control plane minor version when pods cannot be created from 4.20 minor to 4.21 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelNodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled1
Affected runs (7)
Date (UTC)Associated PRProw job
PR #74272108618645423788032
PR #74012108538835301306368
PR #73972108513285430054912
PR #69442108480044224483328
PR #73972108452968792592384
PR #74382108402844888469504
PR #73642108377905737240576
alert [hcp] KubeDeploymentReplicasMismatch fired
full failure pattern:
alert [hcp] KubeDeploymentReplicasMismatch fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days
alert610.17%6 of 59 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 5 daysShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 7 · Oct 6: 7 · Oct 7: 2 · Oct 8: 7 · Oct 9: 6none
Full failure examples (3)
alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-10T00:42:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mchpr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-10T00:43:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mchpr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:21:13Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7317504-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.112:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7317504/providers/microsoft.monitor/accounts/hcps-j7317504", namespace="ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8xgvb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:26:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7317504-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.112:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7317504/providers/microsoft.monitor/accounts/hcps-j7317504", namespace="ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8xgvb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7317504-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="csi-snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.112:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7317504/providers/microsoft.monitor/accounts/hcps-j7317504", namespace="ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8xgvb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster/csi-snapshot-controller has not matched the expected number of replicas for longer than 30 minutes.
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:17:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1936896-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1936896/providers/microsoft.monitor/accounts/hcps-j1936896", namespace="ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frw78", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:17:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1936896-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1936896/providers/microsoft.monitor/accounts/hcps-j1936896", namespace="ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frw78", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:17:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1936896-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1936896/providers/microsoft.monitor/accounts/hcps-j1936896", namespace="ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frw78", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:17:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1936896-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1936896/providers/microsoft.monitor/accounts/hcps-j1936896", namespace="ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frw78", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire6
Affected runs (6)
Date (UTC)Associated PRProw job
PR #74482108692920105177088
PR #69042108611968167317504
PR #74402108607415241936896
PR #74372108513309102706688
PR #72922108438042090934272
PR #71052108409414854643712
alert [svc] BackendControllerQueueDepthHigh fired
full failure pattern:
alert [svc] BackendControllerQueueDepthHigh fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 7 days
alert46.78%4 of 59 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 7 daysShow trend detailsOct 3: 15 · Oct 4: 3 · Oct 5: 26 · Oct 6: 15 · Oct 7: 29 · Oct 8: 19 · Oct 9: 4none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T21:28:57Z
  Ended:       2026-10-09T21:36:57Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8166656-svc", component="backend", name="actualhostedcluster", region="westus3", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T15:56:36Z
  Ended:       2026-10-09T16:18:35Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5582720-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T08:36:10Z
  Ended:       2026-10-09T08:43:12Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2592384-svc", component="backend", name="actualhostedcluster", region="westus3", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #74392108648873588166656
PR #74442108569711095582720
PR #73972108452968792592384
PR #74372108374328641851392
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days
e2e35.08%3 of 59 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 3 daysShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 3 · Oct 8: 4 · Oct 9: 6none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x11708fbe8558>: 
    resource group "patch-tags-qgj9df44rmxf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-qgj9df44rmxf" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dae7a46b-a56b-43d2-8533-c8bfc2bffae7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dae7a46b-a56b-43d2-8533-c8bfc2bffae7",
      "name": "dae7a46b-a56b-43d2-8533-c8bfc2bffae7",
      "status": "Failed",
      "startTime": "2026-10-09T11:53:27.886110615Z",
      "endTime": "2026-10-09T12:17:38.071711819Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb8a5khcdtm17dl9aq7kql2ocecgpqo still exists (deletion dispatched at 2026-10-09T11:53:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x11708fbe8558>: 
    resource group "patch-tags-qgj9df44rmxf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-qgj9df44rmxf" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dae7a46b-a56b-43d2-8533-c8bfc2bffae7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dae7a46b-a56b-43d2-8533-c8bfc2bffae7",
      "name": "dae7a46b-a56b-43d2-8533-c8bfc2bffae7",
      "status": "Failed",
      "startTime": "2026-10-09T11:53:27.886110615Z",
      "endTime": "2026-10-09T12:17:38.071711819Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb8a5khcdtm17dl9aq7kql2ocecgpqo still exists (deletion dispatched at 2026-10-09T11:53:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x330cf0c08078>: 
    resource group "kusto-logs-dxkfj4jt2n4s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-dxkfj4jt2n4s" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/735a1c9a-7d59-48f3-9927-25e5cda56c30
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/735a1c9a-7d59-48f3-9927-25e5cda56c30",
      "name": "735a1c9a-7d59-48f3-9927-25e5cda56c30",
      "status": "Failed",
      "startTime": "2026-10-09T11:55:18.945459438Z",
      "endTime": "2026-10-09T12:19:26.457578493Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb8a54asvaa4eaak0rehljj280alpei still exists (deletion dispatched at 2026-10-09T11:55:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x330cf0c08078>: 
    resource group "kusto-logs-dxkfj4jt2n4s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-dxkfj4jt2n4s" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/735a1c9a-7d59-48f3-9927-25e5cda56c30
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/735a1c9a-7d59-48f3-9927-25e5cda56c30",
      "name": "735a1c9a-7d59-48f3-9927-25e5cda56c30",
      "status": "Failed",
      "startTime": "2026-10-09T11:55:18.945459438Z",
      "endTime": "2026-10-09T12:19:26.457578493Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb8a54asvaa4eaak0rehljj280alpei still exists (deletion dispatched at 2026-10-09T11:55:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2b3aa8d40060>: 
    resource group "private-keyvault-j49nnk79lk7l": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-j49nnk79lk7l" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0feb53b8-520d-463a-b844-2af1832a77a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0feb53b8-520d-463a-b844-2af1832a77a0",
      "name": "0feb53b8-520d-463a-b844-2af1832a77a0",
      "status": "Failed",
      "startTime": "2026-10-09T11:56:12.658651084Z",
      "endTime": "2026-10-09T12:20:16.061081704Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb87h9gah70vts6mbse5o06n385j3o0 still exists (deletion dispatched at 2026-10-09T11:56:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2b3aa8d40060>: 
    resource group "private-keyvault-j49nnk79lk7l": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-j49nnk79lk7l" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0feb53b8-520d-463a-b844-2af1832a77a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0feb53b8-520d-463a-b844-2af1832a77a0",
      "name": "0feb53b8-520d-463a-b844-2af1832a77a0",
      "status": "Failed",
      "startTime": "2026-10-09T11:56:12.658651084Z",
      "endTime": "2026-10-09T12:20:16.061081704Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb87h9gah70vts6mbse5o06n385j3o0 still exists (deletion dispatched at 2026-10-09T11:56:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCreate HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (3)
Date (UTC)Associated PRProw job
PR #74412108512930990395392
PR #71622108511709156085760
PR #74142108377968924430336
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days
e2e35.08%3 of 59 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 3 daysShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 3 · Oct 8: 1 · Oct 9: 5none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x715418a8198>: 
    resource group "rg-hs-pre--d8fntzv77rg5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-6jnw6j" in resourcegroup="rg-hs-pre--d8fntzv77rg5" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f6364b4-168b-4408-93de-ce5f903b20aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f6364b4-168b-4408-93de-ce5f903b20aa",
      "name": "1f6364b4-168b-4408-93de-ce5f903b20aa",
      "status": "Failed",
      "startTime": "2026-10-09T12:05:02.733087351Z",
      "endTime": "2026-10-09T12:29:06.620380667Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb89kg3h59n4o9tks40jdsh5l5jro8g still exists (deletion dispatched at 2026-10-09T12:05:03Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x715418a8198>: 
    resource group "rg-hs-pre--d8fntzv77rg5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-6jnw6j" in resourcegroup="rg-hs-pre--d8fntzv77rg5" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f6364b4-168b-4408-93de-ce5f903b20aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f6364b4-168b-4408-93de-ce5f903b20aa",
      "name": "1f6364b4-168b-4408-93de-ce5f903b20aa",
      "status": "Failed",
      "startTime": "2026-10-09T12:05:02.733087351Z",
      "endTime": "2026-10-09T12:29:06.620380667Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb89kg3h59n4o9tks40jdsh5l5jro8g still exists (deletion dispatched at 2026-10-09T12:05:03Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x52f9b8920c0>: 
    resource group "cni-cilium-lqwdfr6cjvht": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-lqwdfr6cjvht" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d5567703-2f10-4c0b-8cf6-70ff35f674c9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d5567703-2f10-4c0b-8cf6-70ff35f674c9",
      "name": "d5567703-2f10-4c0b-8cf6-70ff35f674c9",
      "status": "Failed",
      "startTime": "2026-10-09T11:57:36.843017136Z",
      "endTime": "2026-10-09T12:21:46.074977182Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb86qasjq198a6vq4pknlkm4b3grgda still exists (deletion dispatched at 2026-10-09T11:57:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x52f9b8920c0>: 
    resource group "cni-cilium-lqwdfr6cjvht": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-lqwdfr6cjvht" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d5567703-2f10-4c0b-8cf6-70ff35f674c9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d5567703-2f10-4c0b-8cf6-70ff35f674c9",
      "name": "d5567703-2f10-4c0b-8cf6-70ff35f674c9",
      "status": "Failed",
      "startTime": "2026-10-09T11:57:36.843017136Z",
      "endTime": "2026-10-09T12:21:46.074977182Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb86qasjq198a6vq4pknlkm4b3grgda still exists (deletion dispatched at 2026-10-09T11:57:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x4a16d2b40f0>: 
    resource group "rg-candidate-5-1-trdpp8hx8tpd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-1-jjf49v" in resourcegroup="rg-candidate-5-1-trdpp8hx8tpd" to finish deleting: GET https://rp.j1851392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/292a5ec0-f28b-48a4-89bc-34c6f69d9a46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/292a5ec0-f28b-48a4-89bc-34c6f69d9a46",
      "name": "292a5ec0-f28b-48a4-89bc-34c6f69d9a46",
      "status": "Failed",
      "startTime": "2026-10-09T03:05:28.202107106Z",
      "endTime": "2026-10-09T03:29:29.34914283Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb0dh1i1h74jbtnnmtcppkg9ambqd8h still exists (deletion dispatched at 2026-10-09T03:05:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x4a16d2b40f0>: 
    resource group "rg-candidate-5-1-trdpp8hx8tpd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-1-jjf49v" in resourcegroup="rg-candidate-5-1-trdpp8hx8tpd" to finish deleting: GET https://rp.j1851392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/292a5ec0-f28b-48a4-89bc-34c6f69d9a46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/292a5ec0-f28b-48a4-89bc-34c6f69d9a46",
      "name": "292a5ec0-f28b-48a4-89bc-34c6f69d9a46",
      "status": "Failed",
      "startTime": "2026-10-09T03:05:28.202107106Z",
      "endTime": "2026-10-09T03:29:29.34914283Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb0dh1i1h74jbtnnmtcppkg9ambqd8h still exists (deletion dispatched at 2026-10-09T03:05:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.11
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a HCP cluster and use cilium CNI plugin1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions1
Affected runs (3)
Date (UTC)Associated PRProw job
PR #74412108512930990395392
PR #71622108511709156085760
PR #74372108374328641851392
alert [svc] MaestroAgentMetricsProxyDown fired
full failure pattern:
alert [svc] MaestroAgentMetricsProxyDown fired
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 5 days
alert35.08%3 of 59 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 5 daysShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 1 · Oct 6: 2 · Oct 7: 1 · Oct 8: 2 · Oct 9: 3none
Full failure examples (3)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T15:02:04Z
  Ended:       2026-10-09T15:12:09Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j4897024-mgmt-1", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T14:26:58Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j5137536-mgmt-1", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T03:29:55Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j9299456-mgmt-2", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert MaestroAgentMetricsProxyDown does not fire3
Affected runs (3)
Date (UTC)Associated PRProw job
PR #73972108558761344897024
PR #67462108552522435137536
PR #73942108385613169299456
alert [hcp] KubePodNotReady fired
full failure pattern:
alert [hcp] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 6 days
alert23.39%2 of 59 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 6 daysShow trend detailsOct 3: 2 · Oct 4: 0 · Oct 5: 3 · Oct 6: 6 · Oct 7: 9 · Oct 8: 3 · Oct 9: 2none
Full failure examples (2)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-10T00:27:09Z
  Ended:       2026-10-10T00:33:08Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8064256-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbj1c9chavhk90qkot47r4fc981jv3j-p7l5u8c0n3u2s1c", pod="router-5869c468cd-cr9j5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbj1c9chavhk90qkot47r4fc981jv3j-p7l5u8c0n3u2s1c/router-5869c468cd-cr9j5 has been in a non-ready state for longer than 5 minutes.
alert fired 94 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-x96nb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-x96nb has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-jtlkd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-jtlkd has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-9zkgt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-9zkgt has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-sp5cx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-sp5cx has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-cls45", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-cls45 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-v5hjd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-v5hjd has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-s8kmb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-s8kmb has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-7wbzr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-7wbzr has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-g6r2m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-g6r2m has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-rq6n8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-rq6n8 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-m29g5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-m29g5 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-pcttg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-pcttg has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-kfj5n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-kfj5n has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-8mwsk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-8mwsk has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-rdzwz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-rdzwz has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-9nrsr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-9nrsr has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-bbf4f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-bbf4f has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-xz7fq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-xz7fq has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-khnm2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-khnm2 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-tn9tb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-tn9tb has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-cks5z", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-cks5z has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-lgjxz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-lgjxz has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-ghjjp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-ghjjp has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-lfpsq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-lfpsq has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-j9pqn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-j9pqn has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:05:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-5tnr6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-5tnr6 has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-zhh5s", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-zhh5s has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-df2m9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-df2m9 has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-d8cbr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-d8cbr has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-q4bf8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-q4bf8 has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-8vkk6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-8vkk6 has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-ltf9m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-ltf9m has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-pqcmf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-pqcmf has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-qddb4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-qddb4 has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-rxqr4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-rxqr4 has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-8x4dn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-8x4dn has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-r4p99", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-r4p99 has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-62kzb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-62kzb has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-rmkv4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-rmkv4 has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-r2blk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-r2blk has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:05:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-mzd6d", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-mzd6d has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-6qk79", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-6qk79 has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-ldl4b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-ldl4b has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-skcks", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-skcks has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-2wnf8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-2wnf8 has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-g9jdz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-g9jdz has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-zqv6n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-zqv6n has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-57999", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-57999 has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-qvtrf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-qvtrf has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-dvwv9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-dvwv9 has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-dncfj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-dncfj has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:05:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-9q52l", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-9q52l has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-5qtxq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-5qtxq has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:05:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-sfjfx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-sfjfx has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-4vqv7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-4vqv7 has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-pvmct", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-pvmct has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-sgbt2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-sgbt2 has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-mtw9g", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-mtw9g has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-nzfck", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-nzfck has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-6n2zh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-6n2zh has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-gl292", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-gl292 has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-ptg2s", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-ptg2s has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-qvtv4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-qvtv4 has been in a non-ready state for longer than 5 minutes.
Firing 64:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-njvdk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-njvdk has been in a non-ready state for longer than 5 minutes.
Firing 65:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-5cqkh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-5cqkh has been in a non-ready state for longer than 5 minutes.
Firing 66:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-t8tsp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-t8tsp has been in a non-ready state for longer than 5 minutes.
Firing 67:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-9mtq6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-9mtq6 has been in a non-ready state for longer than 5 minutes.
Firing 68:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:05:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-qr84n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-qr84n has been in a non-ready state for longer than 5 minutes.
Firing 69:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-nxw6l", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-nxw6l has been in a non-ready state for longer than 5 minutes.
Firing 70:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-rstls", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-rstls has been in a non-ready state for longer than 5 minutes.
Firing 71:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-vgqmv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-vgqmv has been in a non-ready state for longer than 5 minutes.
Firing 72:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-2c8gz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-2c8gz has been in a non-ready state for longer than 5 minutes.
Firing 73:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-9q7pr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-9q7pr has been in a non-ready state for longer than 5 minutes.
Firing 74:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-g7zj8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-g7zj8 has been in a non-ready state for longer than 5 minutes.
Firing 75:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-c9rv5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-c9rv5 has been in a non-ready state for longer than 5 minutes.
Firing 76:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-p6wbm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-p6wbm has been in a non-ready state for longer than 5 minutes.
Firing 77:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-hrfnw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-hrfnw has been in a non-ready state for longer than 5 minutes.
Firing 78:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:05:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-47xk7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-47xk7 has been in a non-ready state for longer than 5 minutes.
Firing 79:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-mspcg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-mspcg has been in a non-ready state for longer than 5 minutes.
Firing 80:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-qn65l", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-qn65l has been in a non-ready state for longer than 5 minutes.
Firing 81:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-v4hhz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-v4hhz has been in a non-ready state for longer than 5 minutes.
Firing 82:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-q4dtm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-q4dtm has been in a non-ready state for longer than 5 minutes.
Firing 83:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-cs2fn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-cs2fn has been in a non-ready state for longer than 5 minutes.
Firing 84:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-tv9qb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-tv9qb has been in a non-ready state for longer than 5 minutes.
Firing 85:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-mtp6n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-mtp6n has been in a non-ready state for longer than 5 minutes.
Firing 86:
  State:       Resolved
  Started:     2026-10-09T15:42:37Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-s249v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-s249v has been in a non-ready state for longer than 5 minutes.
Firing 87:
  State:       Resolved
  Started:     2026-10-09T15:43:35Z
  Ended:       2026-10-09T16:05:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-wmvms", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-wmvms has been in a non-ready state for longer than 5 minutes.
Firing 88:
  State:       Resolved
  Started:     2026-10-09T15:44:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-vrj6w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-vrj6w has been in a non-ready state for longer than 5 minutes.
Firing 89:
  State:       Resolved
  Started:     2026-10-09T15:44:36Z
  Ended:       2026-10-09T16:05:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-zggw8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-zggw8 has been in a non-ready state for longer than 5 minutes.
Firing 90:
  State:       Resolved
  Started:     2026-10-09T15:44:36Z
  Ended:       2026-10-09T16:05:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-5rf8m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-5rf8m has been in a non-ready state for longer than 5 minutes.
Firing 91:
  State:       Resolved
  Started:     2026-10-09T15:44:37Z
  Ended:       2026-10-09T16:05:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-8p7wx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-8p7wx has been in a non-ready state for longer than 5 minutes.
Firing 92:
  State:       Resolved
  Started:     2026-10-09T15:44:37Z
  Ended:       2026-10-09T16:05:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-fvbcc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-fvbcc has been in a non-ready state for longer than 5 minutes.
Firing 93:
  State:       Resolved
  Started:     2026-10-09T15:44:37Z
  Ended:       2026-10-09T16:02:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-jmhw2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-jmhw2 has been in a non-ready state for longer than 5 minutes.
Firing 94:
  State:       Resolved
  Started:     2026-10-09T15:44:37Z
  Ended:       2026-10-09T16:05:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-7t9rz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-7t9rz has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodNotReady does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74392108694594228064256
PR #72482108564560276361216
client rate limiter Wait returned an error: context deadline exceeded
full failure pattern:
client rate limiter Wait returned an error: context deadline exceeded
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days
e2e23.39%2 of 59 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 5 daysShow trend detailsOct 3: 0 · Oct 4: 5 · Oct 5: 36 · Oct 6: 39 · Oct 7: 16 · Oct 8: 0 · Oct 9: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullc5dl28vw558j.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x34f9246db800>: 
    VerifyImagePulled(acrpullc5dl28vw558j.azurecr.io/ubi-minimal) timed out after 2m0.002s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(acrpullc5dl28vw558j.azurecr.io/ubi-minimal) timed out after 2m0.002s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x34f9246db6e0>{
            msg: "failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x34f9246db5e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullc5dl28vw558j.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x34f9246db800>: 
    VerifyImagePulled(acrpullc5dl28vw558j.azurecr.io/ubi-minimal) timed out after 2m0.002s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(acrpullc5dl28vw558j.azurecr.io/ubi-minimal) timed out after 2m0.002s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x34f9246db6e0>{
            msg: "failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x34f9246db5e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullctpjh66q2dr2.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x97759ef0500>: 
    VerifyImagePulled(acrpullctpjh66q2dr2.azurecr.io/ubi-minimal) timed out after 2m0.001s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(acrpullctpjh66q2dr2.azurecr.io/ubi-minimal) timed out after 2m0.001s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x97759ef04a0>{
            msg: "failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x97759f21ec0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullctpjh66q2dr2.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x97759ef0500>: 
    VerifyImagePulled(acrpullctpjh66q2dr2.azurecr.io/ubi-minimal) timed out after 2m0.001s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(acrpullctpjh66q2dr2.azurecr.io/ubi-minimal) timed out after 2m0.001s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x97759ef04a0>{
            msg: "failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x97759f21ec0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #72922108377922371850240
PR #74372108374328641851392
alert [hcp] KubePodCrashLooping fired
full failure pattern:
alert [hcp] KubePodCrashLooping fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 5 days
alert23.39%2 of 59 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 5 daysShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 2 · Oct 6: 5 · Oct 7: 1 · Oct 8: 2 · Oct 9: 2none
Full failure examples (2)
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-10T01:13:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="cloud-network-config-controller-7847666f8b-xtgjl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="78775d9a-b0ca-4c4d-bd49-d00b20e49e68"
  Description: Pod ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/cloud-network-config-controller-7847666f8b-xtgjl (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-10T01:13:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="csi-snapshot-controller-669f7dfb7b-68ftn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6585df9e-c94d-48b0-a688-96a3851ac4dd"
  Description: Pod ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/csi-snapshot-controller-669f7dfb7b-68ftn (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-10T01:14:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="csi-snapshot-controller-669f7dfb7b-68ftn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6585df9e-c94d-48b0-a688-96a3851ac4dd"
  Description: Pod ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/csi-snapshot-controller-669f7dfb7b-68ftn (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-10T01:14:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="cloud-network-config-controller-7847666f8b-xtgjl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="78775d9a-b0ca-4c4d-bd49-d00b20e49e68"
  Description: Pod ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/cloud-network-config-controller-7847666f8b-xtgjl (controller) is in waiting state (reason: "CrashLoopBackOff").
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T07:28:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0934272-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.212:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0934272/providers/microsoft.monitor/accounts/hcps-j0934272", namespace="ocm-arohcpci01-2tb42n0fgkuoc1sge8efonl494ts6f8p-q5w0c9a9b5r6x3n", pod="cloud-network-config-controller-5cf98f8c95-5bd8j", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a135b966-4dab-47fb-8cc2-d75dfe7326f1"
  Description: Pod ocm-arohcpci01-2tb42n0fgkuoc1sge8efonl494ts6f8p-q5w0c9a9b5r6x3n/cloud-network-config-controller-5cf98f8c95-5bd8j (controller) is in waiting state (reason: "CrashLoopBackOff").
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74482108692920105177088
PR #72922108438042090934272
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
full failure pattern:
certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days
e2e23.39%2 of 59 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 3 daysShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 1 · Oct 6: 0 · Oct 7: 1 · Oct 8: 0 · Oct 9: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.008s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xba7a1123110>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.008s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xba7a1123110>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x2741419abce0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x2741419abce0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #72922108438042090934272
PR #74142108377968924430336
alert [svc] KubePodNotReady fired
full failure pattern:
alert [svc] KubePodNotReady fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 6 days
alert11.69%1 of 59 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 6 daysShow trend detailsOct 3: 1 · Oct 4: 0 · Oct 5: 4 · Oct 6: 5 · Oct 7: 2 · Oct 8: 1 · Oct 9: 1none
Full failure examples (1)
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:11:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-23980774-vmss000000-slpqz", region="westus3", severity="warning"
  Description: Pod kube-system/eraser-aks-system-23980774-vmss000000-slpqz has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:11:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-lzv6r", region="westus3", severity="warning"
  Description: Pod kube-system/aks-secrets-store-csi-driver-lzv6r has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:11:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-qbqjf", region="westus3", severity="warning"
  Description: Pod kube-system/ama-metrics-node-qbqjf has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:12:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", namespace="swift-recorder", pod="swift-recorder-6f6gx", region="westus3", severity="warning"
  Description: Pod swift-recorder/swift-recorder-6f6gx has been in a non-ready state for longer than 5 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubePodNotReady does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73972108558761344897024
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedC...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 4 days
e2e11.69%1 of 59 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 4 daysShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 2 · Oct 6: 3 · Oct 7: 0 · Oct 8: 1 · Oct 9: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-2xlsbr with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x2653bb4a2240>: 
    failed to create HCP cluster np-version-upgrade-cluster-2xlsbr: failed waiting for cluster="np-version-upgrade-cluster-2xlsbr" in resourcegroup="rg-np-version-upgrade-2xlsbr-hc74hngnvw7h" to finish creating: GET https://rp.j5021952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c",
      "name": "9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c",
      "status": "Failed",
      "startTime": "2026-10-09T17:24:44.640587758Z",
      "endTime": "2026-10-09T17:53:46.503674235Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-2xlsbr with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x2653bb4a2240>: 
    failed to create HCP cluster np-version-upgrade-cluster-2xlsbr: failed waiting for cluster="np-version-upgrade-cluster-2xlsbr" in resourcegroup="rg-np-version-upgrade-2xlsbr-hc74hngnvw7h" to finish creating: GET https://rp.j5021952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c",
      "name": "9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c",
      "status": "Failed",
      "startTime": "2026-10-09T17:24:44.640587758Z",
      "endTime": "2026-10-09T17:53:46.503674235Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72272108589017825021952
alert [svc] KubeDaemonSetRolloutStuck fired
full failure pattern:
alert [svc] KubeDaemonSetRolloutStuck fired
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 3 days
alert11.69%1 of 59 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 3 daysShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 3 · Oct 6: 3 · Oct 7: 0 · Oct 8: 0 · Oct 9: 1none
Full failure examples (1)
alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:20:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.64.99:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4897024/providers/microsoft.monitor/accounts/services-j4897024", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mfswx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:20:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.64.99:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4897024/providers/microsoft.monitor/accounts/services-j4897024", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mfswx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:20:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.64.99:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4897024/providers/microsoft.monitor/accounts/services-j4897024", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mfswx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:20:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.64.99:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4897024/providers/microsoft.monitor/accounts/services-j4897024", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mfswx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert KubeDaemonSetRolloutStuck does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73972108558761344897024
Deserialization Error: no output from command
full failure pattern:
Deserialization Error: no output from command
Signal: NoiseSignal: Noise — source deserialization/no-output error
e2e11.69%1 of 59 job runs affectedNoiseSignal: Noise — source deserialization/no-output errorShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 1none
Full failure examples (1)
77f90 sp=0x3ae32ce77ed0 pc=0x586c752
net/http.(*Transport).startDialConnForLocked.func1()
	/usr/lib/golang/src/net/http/transport.go:1629 +0x35 fp=0x3ae32ce77fe0 sp=0x3ae32ce77f90 pc=0x586c595
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x3ae32ce77fe8 sp=0x3ae32ce77fe0 pc=0x5540381
created by net/http.(*Transport).startDialConnForLocked in goroutine 119
	/usr/lib/golang/src/net/http/transport.go:1628 +0x112

goroutine 167 gp=0x3ae32ce014a0 m=nil [semacquire]:
runtime.gopark(0x3f00400063cb02?, 0x3ae32da24402?, 0xe0?, 0xc0?, 0x3ae32d696bb8?)
	/usr/lib/golang/src/runtime/proc.go:462 +0xce fp=0x3ae32d6969c0 sp=0x3ae32d6969a0 pc=0x553818e
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:468
runtime.semacquire1(0x16f798a4, 0x0, 0x0, 0x0, 0xd)
	/usr/lib/golang/src/runtime/sema.go:192 +0x232 fp=0x3ae32d696a28 sp=0x3ae32d6969c0 pc=0x5513872
runtime.semacquire(...)
	/usr/lib/golang/src/runtime/sema.go:143
runtime.gcMarkDone()
	/usr/lib/golang/src/runtime/mgc.go:1018 +0x32 fp=0x3ae32d696ac8 sp=0x3ae32d696a28 pc=0x54d6232
runtime.gcAssistAlloc(0x3ae32ce014a0)
	/usr/lib/golang/src/runtime/mgcmark.go:647 +0x2f6 fp=0x3ae32d696b58 sp=0x3ae32d696ac8 pc=0x54dadd6
runtime.deductAssistCredit(0x1?)
	/usr/lib/golang/src/runtime/malloc.go:1861 +0x46 fp=0x3ae32d696b78 sp=0x3ae32d696b58 pc=0x54cb6a6
runtime.mallocgc(0x590, 0xf965760, 0x1)
	/usr/lib/golang/src/runtime/malloc.go:1113 +0x67 fp=0x3ae32d696ba8 sp=0x3ae32d696b78 pc=0x5534e67
runtime.newobject(0x3ae32da2445d?)
	/usr/lib/golang/src/runtime/malloc.go:2157 +0x25 fp=0x3ae32d696bd0 sp=0x3ae32d696ba8 pc=0x54cb7a5
crypto/x509.parseCertificate({0x3ae32da4c000, 0x56d, 0x582})
	/usr/lib/golang/src/crypto/x509/parser.go:887 +0x45 fp=0x3ae32d697068 sp=0x3ae32d696bd0 pc=0x5729285
crypto/x509.ParseCertificate({0x3ae32da4c000?, 0x56d, 0x2ea83?})
	/usr/lib/golang/src/crypto/x509/parser.go:1077 +0x1d fp=0x3ae32d697090 sp=0x3ae32d697068 pc=0x572a69d
crypto/x509.(*CertPool).AppendCertsFromPEM(0x3ae32d4196e0, {0x3ae32da80000?, 0x100a51a0?, 0x10059470?})
	/usr/lib/golang/src/crypto/x509/cert_pool.go:231 +0x125 fp=0x3ae32d6971b0 sp=0x3ae32d697090 pc=0x571d2c5
crypto/x509.loadSystemRoots()
	/usr/lib/golang/src/crypto/x509/root_unix.go:72 +0x391 fp=0x3ae32d697298 sp=0x3ae32d6971b0 pc=0x572ccf1
crypto/x509.initSystemRoots()
	/usr/lib/golang/src/crypto/x509/root.go:43 +0x65 fp=0x3ae32d6972d8 sp=0x3ae32d697298 pc=0x572c785
sync.(*Once).doSlow(0x54ca9ea?, 0x1f1a89e3ef160b9?)
	/usr/lib/golang/src/sync/once.go:78 +0xac fp=0x3ae32d697330 sp=0x3ae32d6972d8 pc=0x555198c
sync.(*Once).Do(...)
	/usr/lib/golang/src/sync/once.go:69
crypto/x509.systemRootsPool()
	/usr/lib/golang/src/crypto/x509/root.go:32 +0x45 fp=0x3ae32d697380 sp=0x3ae32d697330 pc=0x572c645
crypto/x509.(*Certificate).Verify(0x3ae32d4ae008, {{0x3ae32d1d9be0, 0x19}, 0x3ae32d419620, 0x0, {0xc2aa7a68520d364e, 0xb6db3cc, 0x16f76d00}, {0x0, 0x0, ...}, ...})
	/usr/lib/golang/src/crypto/x509/verify.go:578 +0x18b fp=0x3ae32d697530 sp=0x3ae32d697380 pc=0x572f5cb
crypto/tls.(*Conn).verifyServerCertificate(0x3ae32d432008, {0x3ae32d49a5a0, 0x3, 0x4})
	/usr/lib/golang/src/crypto/tls/handshake_client.go:1168 +0xb0e fp=0x3ae32d6978b8 sp=0x3ae32d697530 pc=0x5781dee
crypto/tls.(*clientHandshakeStateTLS13).readServerCertificate(0x3ae32d697c80)
	/usr/lib/golang/src/crypto/tls/handshake_client_tls13.go:640 +0x29a fp=0x3ae32d697ab0 sp=0x3ae32d6978b8 pc=0x578629a
crypto/tls.(*clientHandshakeStateTLS13).handshake(0x3ae32d697c80)
	/usr/lib/golang/src/crypto/tls/handshake_client_tls13.go:141 +0x7cc fp=0x3ae32d697b70 sp=0x3ae32d697ab0 pc=0x578344c
crypto/tls.(*Conn).clientHandshake(0x3ae32d432008, {0x1018a380, 0x3ae32ca24690})
	/usr/lib/golang/src/crypto/tls/handshake_client.go:339 +0x828 fp=0x3ae32d697d88 sp=0x3ae32d697b70 pc=0x577cb68
crypto/tls.(*Conn).clientHandshake-fm({0x1018a380?, 0x3ae32ca24690?})
	<autogenerated>:1 +0x2a fp=0x3ae32d697db0 sp=0x3ae32d697d88 pc=0x57bcaca
crypto/tls.(*Conn).handshakeContext(0x3ae32d432008, {0x1018a380, 0x3ae32c9f4460})
	/usr/lib/golang/src/crypto/tls/conn.go:1562 +0x329 fp=0x3ae32d697f70 sp=0x3ae32d697db0 pc=0x5773829
crypto/tls.(*Conn).HandshakeContext(...)
	/usr/lib/golang/src/crypto/tls/conn.go:1516
net/http.(*persistConn).addTLS.func2()
	/usr/lib/golang/src/net/http/transport.go:1736 +0x6e fp=0x3ae32d697fe0 sp=0x3ae32d697f70 pc=0x586d3ce
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x3ae32d697fe8 sp=0x3ae32d697fe0 pc=0x5540381
created by net/http.(*persistConn).addTLS in goroutine 120
	/usr/lib/golang/src/net/http/transport.go:1732 +0x309

rax    0x0
rbx    0x7fe37a988640
rcx    0x7fe3c265d94c
rdx    0x6
rdi    0xa6f0
rsi    0xa6f4
rbp    0xa6f4
rsp    0x7fe37a987790
r8     0x7fe37a987860
r9     0x7fe3c27834e0
r10    0x8
r11    0x246
r12    0x6
r13    0x7fe37a987b60
r14    0x3ae32bf97860
r15    0xffffffffffffffff
rip    0x7fe3c265d94c
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
77f90 sp=0x3ae32ce77ed0 pc=0x586c752
net/http.(*Transport).startDialConnForLocked.func1()
	/usr/lib/golang/src/net/http/transport.go:1629 +0x35 fp=0x3ae32ce77fe0 sp=0x3ae32ce77f90 pc=0x586c595
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x3ae32ce77fe8 sp=0x3ae32ce77fe0 pc=0x5540381
created by net/http.(*Transport).startDialConnForLocked in goroutine 119
	/usr/lib/golang/src/net/http/transport.go:1628 +0x112

goroutine 167 gp=0x3ae32ce014a0 m=nil [semacquire]:
runtime.gopark(0x3f00400063cb02?, 0x3ae32da24402?, 0xe0?, 0xc0?, 0x3ae32d696bb8?)
	/usr/lib/golang/src/runtime/proc.go:462 +0xce fp=0x3ae32d6969c0 sp=0x3ae32d6969a0 pc=0x553818e
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:468
runtime.semacquire1(0x16f798a4, 0x0, 0x0, 0x0, 0xd)
	/usr/lib/golang/src/runtime/sema.go:192 +0x232 fp=0x3ae32d696a28 sp=0x3ae32d6969c0 pc=0x5513872
runtime.semacquire(...)
	/usr/lib/golang/src/runtime/sema.go:143
runtime.gcMarkDone()
	/usr/lib/golang/src/runtime/mgc.go:1018 +0x32 fp=0x3ae32d696ac8 sp=0x3ae32d696a28 pc=0x54d6232
runtime.gcAssistAlloc(0x3ae32ce014a0)
	/usr/lib/golang/src/runtime/mgcmark.go:647 +0x2f6 fp=0x3ae32d696b58 sp=0x3ae32d696ac8 pc=0x54dadd6
runtime.deductAssistCredit(0x1?)
	/usr/lib/golang/src/runtime/malloc.go:1861 +0x46 fp=0x3ae32d696b78 sp=0x3ae32d696b58 pc=0x54cb6a6
runtime.mallocgc(0x590, 0xf965760, 0x1)
	/usr/lib/golang/src/runtime/malloc.go:1113 +0x67 fp=0x3ae32d696ba8 sp=0x3ae32d696b78 pc=0x5534e67
runtime.newobject(0x3ae32da2445d?)
	/usr/lib/golang/src/runtime/malloc.go:2157 +0x25 fp=0x3ae32d696bd0 sp=0x3ae32d696ba8 pc=0x54cb7a5
crypto/x509.parseCertificate({0x3ae32da4c000, 0x56d, 0x582})
	/usr/lib/golang/src/crypto/x509/parser.go:887 +0x45 fp=0x3ae32d697068 sp=0x3ae32d696bd0 pc=0x5729285
crypto/x509.ParseCertificate({0x3ae32da4c000?, 0x56d, 0x2ea83?})
	/usr/lib/golang/src/crypto/x509/parser.go:1077 +0x1d fp=0x3ae32d697090 sp=0x3ae32d697068 pc=0x572a69d
crypto/x509.(*CertPool).AppendCertsFromPEM(0x3ae32d4196e0, {0x3ae32da80000?, 0x100a51a0?, 0x10059470?})
	/usr/lib/golang/src/crypto/x509/cert_pool.go:231 +0x125 fp=0x3ae32d6971b0 sp=0x3ae32d697090 pc=0x571d2c5
crypto/x509.loadSystemRoots()
	/usr/lib/golang/src/crypto/x509/root_unix.go:72 +0x391 fp=0x3ae32d697298 sp=0x3ae32d6971b0 pc=0x572ccf1
crypto/x509.initSystemRoots()
	/usr/lib/golang/src/crypto/x509/root.go:43 +0x65 fp=0x3ae32d6972d8 sp=0x3ae32d697298 pc=0x572c785
sync.(*Once).doSlow(0x54ca9ea?, 0x1f1a89e3ef160b9?)
	/usr/lib/golang/src/sync/once.go:78 +0xac fp=0x3ae32d697330 sp=0x3ae32d6972d8 pc=0x555198c
sync.(*Once).Do(...)
	/usr/lib/golang/src/sync/once.go:69
crypto/x509.systemRootsPool()
	/usr/lib/golang/src/crypto/x509/root.go:32 +0x45 fp=0x3ae32d697380 sp=0x3ae32d697330 pc=0x572c645
crypto/x509.(*Certificate).Verify(0x3ae32d4ae008, {{0x3ae32d1d9be0, 0x19}, 0x3ae32d419620, 0x0, {0xc2aa7a68520d364e, 0xb6db3cc, 0x16f76d00}, {0x0, 0x0, ...}, ...})
	/usr/lib/golang/src/crypto/x509/verify.go:578 +0x18b fp=0x3ae32d697530 sp=0x3ae32d697380 pc=0x572f5cb
crypto/tls.(*Conn).verifyServerCertificate(0x3ae32d432008, {0x3ae32d49a5a0, 0x3, 0x4})
	/usr/lib/golang/src/crypto/tls/handshake_client.go:1168 +0xb0e fp=0x3ae32d6978b8 sp=0x3ae32d697530 pc=0x5781dee
crypto/tls.(*clientHandshakeStateTLS13).readServerCertificate(0x3ae32d697c80)
	/usr/lib/golang/src/crypto/tls/handshake_client_tls13.go:640 +0x29a fp=0x3ae32d697ab0 sp=0x3ae32d6978b8 pc=0x578629a
crypto/tls.(*clientHandshakeStateTLS13).handshake(0x3ae32d697c80)
	/usr/lib/golang/src/crypto/tls/handshake_client_tls13.go:141 +0x7cc fp=0x3ae32d697b70 sp=0x3ae32d697ab0 pc=0x578344c
crypto/tls.(*Conn).clientHandshake(0x3ae32d432008, {0x1018a380, 0x3ae32ca24690})
	/usr/lib/golang/src/crypto/tls/handshake_client.go:339 +0x828 fp=0x3ae32d697d88 sp=0x3ae32d697b70 pc=0x577cb68
crypto/tls.(*Conn).clientHandshake-fm({0x1018a380?, 0x3ae32ca24690?})
	<autogenerated>:1 +0x2a fp=0x3ae32d697db0 sp=0x3ae32d697d88 pc=0x57bcaca
crypto/tls.(*Conn).handshakeContext(0x3ae32d432008, {0x1018a380, 0x3ae32c9f4460})
	/usr/lib/golang/src/crypto/tls/conn.go:1562 +0x329 fp=0x3ae32d697f70 sp=0x3ae32d697db0 pc=0x5773829
crypto/tls.(*Conn).HandshakeContext(...)
	/usr/lib/golang/src/crypto/tls/conn.go:1516
net/http.(*persistConn).addTLS.func2()
	/usr/lib/golang/src/net/http/transport.go:1736 +0x6e fp=0x3ae32d697fe0 sp=0x3ae32d697f70 pc=0x586d3ce
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x3ae32d697fe8 sp=0x3ae32d697fe0 pc=0x5540381
created by net/http.(*persistConn).addTLS in goroutine 120
	/usr/lib/golang/src/net/http/transport.go:1732 +0x309

rax    0x0
rbx    0x7fe37a988640
rcx    0x7fe3c265d94c
rdx    0x6
rdi    0xa6f0
rsi    0xa6f4
rbp    0xa6f4
rsp    0x7fe37a987790
r8     0x7fe37a987860
r9     0x7fe3c27834e0
r10    0x8
r11    0x246
r12    0x6
r13    0x7fe37a987b60
r14    0x3ae32bf97860
r15    0xffffffffffffffff
rip    0x7fe3c265d94c
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should not be able to perform various invalid operations on cluster resources1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74462108673315018117120
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; provider Microsoft.RedHatOpenShift
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)
e2e58.47%5 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 3 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1 · Oct 8: 1 · Oct 9: 8none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x87a272e80c0>: 
    resource group "autoscaling-cluster-dch49t4bvhmd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-dch49t4bvhmd" to finish deleting: GET https://rp.j2706688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/314e6611-87cc-467f-b3e8-5c7bded86eeb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/314e6611-87cc-467f-b3e8-5c7bded86eeb",
      "name": "314e6611-87cc-467f-b3e8-5c7bded86eeb",
      "status": "Failed",
      "startTime": "2026-10-09T12:11:02.340187908Z",
      "endTime": "2026-10-09T12:35:03.752595501Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x87a272e80c0>: 
    resource group "autoscaling-cluster-dch49t4bvhmd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-dch49t4bvhmd" to finish deleting: GET https://rp.j2706688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/314e6611-87cc-467f-b3e8-5c7bded86eeb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/314e6611-87cc-467f-b3e8-5c7bded86eeb",
      "name": "314e6611-87cc-467f-b3e8-5c7bded86eeb",
      "status": "Failed",
      "startTime": "2026-10-09T12:11:02.340187908Z",
      "endTime": "2026-10-09T12:35:03.752595501Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3f620761c090>: 
    resource group "customer-rg-lbqh9vzm8skr": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-lbqh9vzm8skr" to finish deleting: GET https://rp.j2706688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff9dce59-52f3-4407-ad07-f8872d204253
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff9dce59-52f3-4407-ad07-f8872d204253",
      "name": "ff9dce59-52f3-4407-ad07-f8872d204253",
      "status": "Failed",
      "startTime": "2026-10-09T12:06:09.323431375Z",
      "endTime": "2026-10-09T12:30:13.733939626Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3f620761c090>: 
    resource group "customer-rg-lbqh9vzm8skr": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-lbqh9vzm8skr" to finish deleting: GET https://rp.j2706688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff9dce59-52f3-4407-ad07-f8872d204253
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff9dce59-52f3-4407-ad07-f8872d204253",
      "name": "ff9dce59-52f3-4407-ad07-f8872d204253",
      "status": "Failed",
      "startTime": "2026-10-09T12:06:09.323431375Z",
      "endTime": "2026-10-09T12:30:13.733939626Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1bf97134030>: 
    resource group "arm64-vm-cluster-rvxwb88jbj7x": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-rvxwb88jbj7x" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2a654fd8-0468-45a6-8d9a-ee003fb30146
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2a654fd8-0468-45a6-8d9a-ee003fb30146",
      "name": "2a654fd8-0468-45a6-8d9a-ee003fb30146",
      "status": "Failed",
      "startTime": "2026-10-09T11:58:55.836253279Z",
      "endTime": "2026-10-09T12:22:56.194703206Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1bf97134030>: 
    resource group "arm64-vm-cluster-rvxwb88jbj7x": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-rvxwb88jbj7x" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2a654fd8-0468-45a6-8d9a-ee003fb30146
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2a654fd8-0468-45a6-8d9a-ee003fb30146",
      "name": "2a654fd8-0468-45a6-8d9a-ee003fb30146",
      "status": "Failed",
      "startTime": "2026-10-09T11:58:55.836253279Z",
      "endTime": "2026-10-09T12:22:56.194703206Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (7)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group2
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should be able to log into a cluster via a breakglass session1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelSRE should return 409 when boot diagnostics is disabled on a VM1
Affected runs (5)
Date (UTC)Associated PRProw job
PR #74372108513309102706688
PR #74412108512930990395392
PR #71622108511709156085760
PR #74142108377968924430336
PR #74372108374328641851392
cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
full failure pattern:
cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
e2e46.78%4 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-rf2ccpxh56qx/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-rf2ccpxh56qx/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-8jfp6hmxz2vs/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-8jfp6hmxz2vs/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-tq5mmp4dz9tr/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-tq5mmp4dz9tr/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation4
Affected runs (4)
Date (UTC)Associated PRProw job
PR #72482108661352737280000
PR #74442108611234440941568
PR #74402108607415241936896
PR #74372108452755680006144
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: nodePools, serviceProviderCluster...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: nodePools, serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior history
e2e35.08%3 of 59 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 2 day(s) — New failure pattern — no prior historyShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1 · Oct 8: 0 · Oct 9: 4none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1d94d9c80a8>: 
    resource group "autoscaling-cluster-tj5rdtq5zbg6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-tj5rdtq5zbg6" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/962a68bc-b321-487b-b276-789b2307a373
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/962a68bc-b321-487b-b276-789b2307a373",
      "name": "962a68bc-b321-487b-b276-789b2307a373",
      "status": "Failed",
      "startTime": "2026-10-09T11:58:48.91579633Z",
      "endTime": "2026-10-09T12:22:56.421671364Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1d94d9c80a8>: 
    resource group "autoscaling-cluster-tj5rdtq5zbg6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-tj5rdtq5zbg6" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/962a68bc-b321-487b-b276-789b2307a373
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/962a68bc-b321-487b-b276-789b2307a373",
      "name": "962a68bc-b321-487b-b276-789b2307a373",
      "status": "Failed",
      "startTime": "2026-10-09T11:58:48.91579633Z",
      "endTime": "2026-10-09T12:22:56.421671364Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3346ee20e4b0>: 
    resource group "feature-aggregation-vk8bg8lddcq2": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-vk8bg8lddcq2" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4724c843-31f5-4e70-a53a-a9dc39567964
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4724c843-31f5-4e70-a53a-a9dc39567964",
      "name": "4724c843-31f5-4e70-a53a-a9dc39567964",
      "status": "Failed",
      "startTime": "2026-10-09T12:05:11.356025969Z",
      "endTime": "2026-10-09T12:29:15.890678807Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3346ee20e4b0>: 
    resource group "feature-aggregation-vk8bg8lddcq2": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-vk8bg8lddcq2" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4724c843-31f5-4e70-a53a-a9dc39567964
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4724c843-31f5-4e70-a53a-a9dc39567964",
      "name": "4724c843-31f5-4e70-a53a-a9dc39567964",
      "status": "Failed",
      "startTime": "2026-10-09T12:05:11.356025969Z",
      "endTime": "2026-10-09T12:29:15.890678807Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x10e954cac0c0>: 
    resource group "fips-enabled-g648xvh6nzj9": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-g648xvh6nzj9" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6fa493d6-7fae-4369-b969-8577dd6dfc89
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6fa493d6-7fae-4369-b969-8577dd6dfc89",
      "name": "6fa493d6-7fae-4369-b969-8577dd6dfc89",
      "status": "Failed",
      "startTime": "2026-10-09T11:52:00.540461838Z",
      "endTime": "2026-10-09T12:16:05.840029374Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x10e954cac0c0>: 
    resource group "fips-enabled-g648xvh6nzj9": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-g648xvh6nzj9" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6fa493d6-7fae-4369-b969-8577dd6dfc89
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6fa493d6-7fae-4369-b969-8577dd6dfc89",
      "name": "6fa493d6-7fae-4369-b969-8577dd6dfc89",
      "status": "Failed",
      "startTime": "2026-10-09T11:52:00.540461838Z",
      "endTime": "2026-10-09T12:16:05.840029374Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (4)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.01
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelFIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property1
Affected runs (3)
Date (UTC)Associated PRProw job
PR #74412108512930990395392
PR #71622108511709156085760
PR #74372108374328641851392
alert [svc] LeaderElectionLeaseStale fired
full failure pattern:
alert [svc] LeaderElectionLeaseStale fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)
alert23.39%2 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 5 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 5 · Oct 6: 6 · Oct 7: 3 · Oct 8: 6 · Oct 9: 2none
Full failure examples (2)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-10T00:52:27Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j9187456-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.0:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9187456/providers/microsoft.monitor/accounts/services-j9187456", namespace="klusterlet-2tbj2sv7sdukmvi25teg7m48qmbbnkqf", pod="arohcp-monitor-kube-state-metrics-7988dc648c-ns4ml", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tbj2sv7sdukmvi25teg7m48qmbbnkqf on cluster ci01-j9187456-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T23:35:06Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j8117120-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.216:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8117120/providers/microsoft.monitor/accounts/services-j8117120", namespace="klusterlet-2tbhpus9mi4ns169dgekasr6fitd43j9", pod="arohcp-monitor-kube-state-metrics-7988dc648c-bc7zx", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tbhpus9mi4ns169dgekasr6fitd43j9 on cluster ci01-j8117120-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert LeaderElectionLeaseStale does not fire2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #72272108694589719187456
PR #74462108673315018117120
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServic...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)
e2e23.39%2 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 4 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 2 · Oct 6: 1 · Oct 7: 0 · Oct 8: 2 · Oct 9: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x126a68f88048>: 
    resource group "nodepool-update-nodes-ppzw4fscb65q": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-ppzw4fscb65q" to finish deleting: GET https://rp.j7543168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5402a6f6-777b-405f-bf42-17f6202e880d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5402a6f6-777b-405f-bf42-17f6202e880d",
      "name": "5402a6f6-777b-405f-bf42-17f6202e880d",
      "status": "Failed",
      "startTime": "2026-10-09T09:00:29.162024664Z",
      "endTime": "2026-10-09T09:24:36.108080611Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb5gs6ackm9riurrcuq3gadk7un22q7 still exists (deletion dispatched at 2026-10-09T09:00:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x126a68f88048>: 
    resource group "nodepool-update-nodes-ppzw4fscb65q": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-ppzw4fscb65q" to finish deleting: GET https://rp.j7543168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5402a6f6-777b-405f-bf42-17f6202e880d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5402a6f6-777b-405f-bf42-17f6202e880d",
      "name": "5402a6f6-777b-405f-bf42-17f6202e880d",
      "status": "Failed",
      "startTime": "2026-10-09T09:00:29.162024664Z",
      "endTime": "2026-10-09T09:24:36.108080611Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb5gs6ackm9riurrcuq3gadk7un22q7 still exists (deletion dispatched at 2026-10-09T09:00:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2f3c0aa90108>: 
    resource group "admin-credential-lifecycle-test-lxkxdsm95v5q": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-xck96d" in resourcegroup="admin-credential-lifecycle-test-lxkxdsm95v5q" to finish deleting: GET https://rp.j4643712.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d7634497-b76b-417d-ba61-d17625706ddd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d7634497-b76b-417d-ba61-d17625706ddd",
      "name": "d7634497-b76b-417d-ba61-d17625706ddd",
      "status": "Failed",
      "startTime": "2026-10-09T05:12:01.285111592Z",
      "endTime": "2026-10-09T05:36:09.092898944Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb2dcbc75fd0t83s3l86makdqbo52ni still exists (deletion dispatched at 2026-10-09T05:12:02Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2f3c0aa90108>: 
    resource group "admin-credential-lifecycle-test-lxkxdsm95v5q": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-xck96d" in resourcegroup="admin-credential-lifecycle-test-lxkxdsm95v5q" to finish deleting: GET https://rp.j4643712.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d7634497-b76b-417d-ba61-d17625706ddd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d7634497-b76b-417d-ba61-d17625706ddd",
      "name": "d7634497-b76b-417d-ba61-d17625706ddd",
      "status": "Failed",
      "startTime": "2026-10-09T05:12:01.285111592Z",
      "endTime": "2026-10-09T05:36:09.092898944Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb2dcbc75fd0t83s3l86makdqbo52ni still exists (deletion dispatched at 2026-10-09T05:12:02Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update nodepool replicas and autoscaling1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74392108458935697543168
PR #71052108409414854643712
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): co...
full failure pattern:
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e23.39%2 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3475813fe160>: 
    DNS for route host agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x3475813fe0e0>{
            msg: "DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3475813fe160>: 
    DNS for route host agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x3475813fe0e0>{
            msg: "DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x247e648e4440>: 
    DNS for route host agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x247e648e4420>{
            msg: "DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x247e648e4440>: 
    DNS for route host agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x247e648e4420>{
            msg: "DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to retrieve cluster and nodepool status active versions2
Affected runs (2)
Date (UTC)Associated PRProw job
PR #71652108676067857272832
PR #74392108648873588166656
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline...
full failure pattern:
DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e23.39%2 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-j7bk7p"
Unexpected error:
    <*fmt.wrapError | 0x76fa41aa1e0>: 
    DNS for route host agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x76fa41aa1a0>{
            msg: "DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-j7bk7p"
Unexpected error:
    <*fmt.wrapError | 0x76fa41aa1e0>: 
    DNS for route host agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x76fa41aa1a0>{
            msg: "DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-w8jdcn"
Unexpected error:
    <*fmt.wrapError | 0xd9c285fc740>: 
    DNS for route host agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0xd9c285fc6e0>{
            msg: "DNS for agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-w8jdcn"
Unexpected error:
    <*fmt.wrapError | 0xd9c285fc740>: 
    DNS for route host agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0xd9c285fc6e0>{
            msg: "DNS for agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.211
Affected runs (2)
Date (UTC)Associated PRProw job
PR #72272108648932484583424
PR #74372108452755680006144
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantRes...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
e2e23.39%2 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3c8ada036198>: 
    resource group "rg-candidate-4-22-7xk4vqr4kmpg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-68b4x4" in resourcegroup="rg-candidate-4-22-7xk4vqr4kmpg" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/25b3de70-92ee-47c8-9155-8c0ca5a96889
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/25b3de70-92ee-47c8-9155-8c0ca5a96889",
      "name": "25b3de70-92ee-47c8-9155-8c0ca5a96889",
      "status": "Failed",
      "startTime": "2026-10-09T12:01:27.302551031Z",
      "endTime": "2026-10-09T12:25:36.409770296Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb89kods8necs5a4oar2jo8ji1411tb still exists (deletion dispatched at 2026-10-09T12:01:28Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3c8ada036198>: 
    resource group "rg-candidate-4-22-7xk4vqr4kmpg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-68b4x4" in resourcegroup="rg-candidate-4-22-7xk4vqr4kmpg" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/25b3de70-92ee-47c8-9155-8c0ca5a96889
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/25b3de70-92ee-47c8-9155-8c0ca5a96889",
      "name": "25b3de70-92ee-47c8-9155-8c0ca5a96889",
      "status": "Failed",
      "startTime": "2026-10-09T12:01:27.302551031Z",
      "endTime": "2026-10-09T12:25:36.409770296Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb89kods8necs5a4oar2jo8ji1411tb still exists (deletion dispatched at 2026-10-09T12:01:28Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3172298ee060>: 
    resource group "customer-rg-85nx9txxtvjg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-85nx9txxtvjg" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c208ad27-e494-4933-aedb-2b311ea04677
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c208ad27-e494-4933-aedb-2b311ea04677",
      "name": "c208ad27-e494-4933-aedb-2b311ea04677",
      "status": "Failed",
      "startTime": "2026-10-09T11:49:09.861950907Z",
      "endTime": "2026-10-09T12:13:16.248989406Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb87377q9r4c48pr0vofden0ac6k3ii still exists (deletion dispatched at 2026-10-09T11:49:10Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3172298ee060>: 
    resource group "customer-rg-85nx9txxtvjg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-85nx9txxtvjg" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c208ad27-e494-4933-aedb-2b311ea04677
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c208ad27-e494-4933-aedb-2b311ea04677",
      "name": "c208ad27-e494-4933-aedb-2b311ea04677",
      "status": "Failed",
      "startTime": "2026-10-09T11:49:09.861950907Z",
      "endTime": "2026-10-09T12:13:16.248989406Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb87377q9r4c48pr0vofden0ac6k3ii still exists (deletion dispatched at 2026-10-09T11:49:10Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #74412108512930990395392
PR #71622108511709156085760
failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1
full failure pattern:
failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
provision23.39%2 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 2none
Full failure examples (2)
time=2026-10-09T16:49:10.013Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 description="Step mitigations\n Kind: Helm\n"
time=2026-10-09T16:49:10.529Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 values="map[azureCnsCpuLimits:map[enabled:true] jobImage:mcr.microsoft.com/aks/command/runtime@sha256:37df117f1179d78fd372fd35878d53ff30dfb350fb571fb197f3a2bee48bd157]"
time=2026-10-09T16:49:10.529Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1
time=2026-10-09T16:49:11.005Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 namespace=aro-hcp-mitigations
time=2026-10-09T16:49:11.316Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 dryRun=false
time=2026-10-09T16:49:11.316Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1
time=2026-10-09T16:49:11.444Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1
time=2026-10-09T16:55:10.359Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 err="failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-09T16:55:10.360Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-09T15:19:24.628Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 description="Step mitigations\n Kind: Helm\n"
time=2026-10-09T15:19:25.146Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 values="map[azureCnsCpuLimits:map[enabled:true] jobImage:mcr.microsoft.com/aks/command/runtime@sha256:37df117f1179d78fd372fd35878d53ff30dfb350fb571fb197f3a2bee48bd157]"
time=2026-10-09T15:19:25.146Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-09T15:19:25.687Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 namespace=aro-hcp-mitigations
time=2026-10-09T15:19:25.942Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 dryRun=false
time=2026-10-09T15:19:25.942Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-09T15:19:26.093Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-09T15:25:12.869Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 err="failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-09T15:25:12.869Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"
Contributing tests (2)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/mitigations (stamp=1)1
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/mitigations (stamp=2)1
Affected runs (2)
Date (UTC)Associated PRProw job
PR #71332108594476522934272
PR #71332108567720399212544
ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement
full failure pattern:
ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
alert11.69%1 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 4 · Oct 9: 2none
Full failure examples (2)
alert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6482688/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j6482688' could not be found."
  }
}
--------------------------------------------------------------------------------
alert collection incomplete
failed to fetch svc alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6482688/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j6482688' could not be found."
  }
}
--------------------------------------------------------------------------------
Contributing tests (2)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [hcp] alert collection is complete1
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert collection is complete1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72262108491565306482688
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)
e2e11.69%1 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 2 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 1 · Oct 8: 0 · Oct 9: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x14de9654a318>: 
    resource group "arm64-vm-cluster-bbmplmj7xnjh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-bbmplmj7xnjh" to finish deleting: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8789fff8-eb02-48e7-adfa-7a5bedc2a9ed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8789fff8-eb02-48e7-adfa-7a5bedc2a9ed",
      "name": "8789fff8-eb02-48e7-adfa-7a5bedc2a9ed",
      "status": "Failed",
      "startTime": "2026-10-09T12:10:18.071494213Z",
      "endTime": "2026-10-09T12:34:22.504221789Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x14de9654a318>: 
    resource group "arm64-vm-cluster-bbmplmj7xnjh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-bbmplmj7xnjh" to finish deleting: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8789fff8-eb02-48e7-adfa-7a5bedc2a9ed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8789fff8-eb02-48e7-adfa-7a5bedc2a9ed",
      "name": "8789fff8-eb02-48e7-adfa-7a5bedc2a9ed",
      "status": "Failed",
      "startTime": "2026-10-09T12:10:18.071494213Z",
      "endTime": "2026-10-09T12:34:22.504221789Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xdad5eeb8138>: 
    resource group "external-auth-cluster-cmj7tlkjf2f4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-cmj7tlkjf2f4" to finish deleting: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a7900d0f-9832-43df-824c-157a3931804e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a7900d0f-9832-43df-824c-157a3931804e",
      "name": "a7900d0f-9832-43df-824c-157a3931804e",
      "status": "Failed",
      "startTime": "2026-10-09T12:11:18.997805773Z",
      "endTime": "2026-10-09T12:35:21.856574986Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xdad5eeb8138>: 
    resource group "external-auth-cluster-cmj7tlkjf2f4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-cmj7tlkjf2f4" to finish deleting: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a7900d0f-9832-43df-824c-157a3931804e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a7900d0f-9832-43df-824c-157a3931804e",
      "name": "a7900d0f-9832-43df-824c-157a3931804e",
      "status": "Failed",
      "startTime": "2026-10-09T12:11:18.997805773Z",
      "endTime": "2026-10-09T12:35:21.856574986Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with an external auth config and get the external auth config1
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create node pools with ARM64-based VMs1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73972108513285430054912
timeout during CreateHCPClusterAndWait; context deadline exceeded
full failure pattern:
timeout during CreateHCPClusterAndWait; context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e11.69%1 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 2none
Full failure examples (2)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x13eb0e2191a0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-bkn7fgqj44mp" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-bkn7fgqj44mp\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x13eb0e00ddf0>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x13eb0e2191a0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-bkn7fgqj44mp" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-bkn7fgqj44mp\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x13eb0e00ddf0>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapErrors | 0x7f2f57b95f0>: 
    failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-hwcbq22lzw9l" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-hwcbq22lzw9l\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x7f2f606c4c0>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l",
            },
            <*fmt.wrapErrors | 0x7f2f57b95c0>{
                msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-hwcbq22lzw9l\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0x7f2f606c4c0>{
                        s: "timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapErrors | 0x7f2f57b95f0>: 
    failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-hwcbq22lzw9l" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-hwcbq22lzw9l\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x7f2f606c4c0>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l",
            },
            <*fmt.wrapErrors | 0x7f2f57b95c0>{
                msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-hwcbq22lzw9l\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0x7f2f606c4c0>{
                        s: "timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
Contributing tests (2)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create an HCP cluster with back-level version 4.201
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to update node pool labels and taints1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74402108607415241936896
alert [svc] FrontendPathLatency fired
full failure pattern:
alert [svc] FrontendPathLatency fired
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)
alert11.69%1 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 2 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 1 · Oct 9: 1none
Full failure examples (1)
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T10:08:14Z
  Ended:       2026-10-09T10:15:16Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j4483328-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: More than 5% of frontend requests for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j4483328-svc).
Firing 2:
  State:       Resolved
  Started:     2026-10-09T10:31:13Z
  Ended:       2026-10-09T10:40:15Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j4483328-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: More than 5% of frontend requests for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j4483328-svc).
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert FrontendPathLatency does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #69442108480044224483328
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
full failure pattern:
Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e11.69%1 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to test admin credentials before cluster ready, then full admin credential lifecycle1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74402108607415241936896
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hype...
full failure pattern:
ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 5 day(s) — New failure pattern — no prior history
e2e11.69%1 of 59 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 5 day(s) — New failure pattern — no prior historyShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 1 · Oct 6: 2 · Oct 7: 1 · Oct 8: 2 · Oct 9: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:495]: failed to create node pool
Unexpected error:
    <*fmt.wrapError | 0x19713c8ce520>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "acr-pull" in resourcegroup="acr-pull-dfz6ss2tmqmt" to finish creating: GET https://rp.j0006144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5c0e150d-aecf-4c68-b9dd-c7ae7dbf7400
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5c0e150d-aecf-4c68-b9dd-c7ae7dbf7400",
      "name": "5c0e150d-aecf-4c68-b9dd-c7ae7dbf7400",
      "status": "Failed",
      "startTime": "2026-10-09T08:12:53.421461459Z",
      "endTime": "2026-10-09T08:42:02.147015674Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:495]: failed to create node pool
Unexpected error:
    <*fmt.wrapError | 0x19713c8ce520>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "acr-pull" in resourcegroup="acr-pull-dfz6ss2tmqmt" to finish creating: GET https://rp.j0006144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5c0e150d-aecf-4c68-b9dd-c7ae7dbf7400
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5c0e150d-aecf-4c68-b9dd-c7ae7dbf7400",
      "name": "5c0e150d-aecf-4c68-b9dd-c7ae7dbf7400",
      "status": "Failed",
      "startTime": "2026-10-09T08:12:53.421461459Z",
      "endTime": "2026-10-09T08:42:02.147015674Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to create a cluster with ACR pull via managed identity and pull from a private ACR1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74372108452755680006144
ERROR CODE: ResourceGroupNotFound; provider Microsoft.Insights
full failure pattern:
ERROR CODE: ResourceGroupNotFound; provider Microsoft.Insights
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)
alert11.69%1 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 2 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 2 · Oct 9: 1none
Full failure examples (1)
alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6482688: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6482688/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j6482688' could not be found."
  }
}
--------------------------------------------------------------------------------
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [infra] alert collection is complete1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72262108491565306482688
ERROR CODE: DeploymentNotFound; provider Microsoft.ContainerService
full failure pattern:
ERROR CODE: DeploymentNotFound; provider Microsoft.ContainerService
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
provision11.69%1 of 59 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 1none
Full failure examples (1)
time=2026-10-09T22:05:07.216Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-09T22:06:09.182Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-09T22:05:07.225651655Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7960192-mgmt-2\",\"clusterName\":\"ci01-j7960192-mgmt-2\",\"region\":\"westus3\"}\n{\"time\":\"2026-10-09T22:05:08.960792766Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7960192-mgmt-2\",\"clusterName\":\"ci01-j7960192-mgmt-2\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-09T22:05:09.292773856Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7960192-mgmt-2\",\"clusterName\":\"ci01-j7960192-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-10-09T22:06:09.178268762Z\",\"level\":\"ERROR\",\"msg\":\"collecting deployment operations\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7960192-mgmt-2\",\"clusterName\":\"ci01-j7960192-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"err\":{\"errorCode\":\"DeploymentNotFound\",\"statusCode\":404,\"errorMessage\":\"GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7960192-mgmt-2/deployments/aks-cluster-create/operations\\n--------------------------------------------------------------------------------\\nRESPONSE 404: 404 Not Found\\nERROR CODE: DeploymentNotFound\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentNotFound\\\",\\n    \\\"message\\\": \\\"Deployment 'aks-cluster-create' could not be found.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"}}\n{\"time\":\"2026-10-09T22:06:09.178359704Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7960192-mgmt-2\",\"clusterName\":\"ci01-j7960192-mgmt-2\",\"region\":\"westus3\",\"err\":\"submitting deployment aks-cluster-create: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7960192-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\n--------------------------------------------------------------------------------\\nRESPONSE 400: 400 Bad Request\\nERROR CODE: InvalidTemplateDeployment\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"InvalidTemplateDeployment\\\",\\n    \\\"message\\\": \\\"The template deployment 'aks-cluster-create' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.ContainerService/managedClusters (2026-04-02-preview)' reported preflight validation errors. Tracking id is 'af747446-bfb7-43e9-a875-3f8953ec8e67'. See inner errors for details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ListResourceSKUsFailed\\\",\\n        \\\"message\\\": \\\"Preflight validation check for resource(s) for container service ci01-j7960192-mgmt-2 in resource group hcp-underlay-ci01-j7960192-mgmt-2 failed. Message: List resource SKUs in subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX failed. Details: \\\"\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"1m1.952725139s\"}\n exit status 1"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2)1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72492108674180797960192
alert [svc] FleetControllerRetryHotLoop fired
full failure pattern:
alert [svc] FleetControllerRetryHotLoop fired
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
alert11.69%1 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T08:14:32Z
  Ended:       2026-10-09T08:32:31Z
  Severity:    Sev3
  Labels:      alertname="FleetControllerRetryHotLoop", cluster="ci01-j0006144-svc", component="fleet", name="nodepoolcontroller", region="westus3", severity="warning"
  Description: Fleet controller workqueue nodepoolcontroller has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert FleetControllerRetryHotLoop does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74372108452755680006144
alert [svc] userJourneyClusterProvisionErrors6h30m fired
full failure pattern:
alert [svc] userJourneyClusterProvisionErrors6h30m fired
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
alert11.69%1 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:03:21Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterProvisionErrors6h30m", cluster="ci01-j1936896-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1936896/providers/microsoft.monitor/accounts/services-j1936896", region="westus3", severity="3", short_window="30m", slo="cluster-provision-errors"
  Description: More than 30% of completed cluster create (install) operations on ci01-j1936896-svc failed over the last 6 hours (at least 5 completions), a 6x burn of the 95% SLO budget.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert userJourneyClusterProvisionErrors6h30m does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74402108607415241936896
query-basic: command failed: exit status 1
full failure pattern:
query-basic: command failed: exit status 1
Signal: IndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior history
e2e11.69%1 of 59 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 1 day(s) — New failure pattern — no prior historyShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 651.764s.
Expected success, but got an error:
    <*errors.errorString | 0x104322d955a0>: 
    must-gather CLI test failures:
      query-basic: command failed: exit status 1
    output: �[37m[23:00:07.094]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452"
    }�[0m
    �[37m[23:00:10.068]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:08.6161084Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\", ActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", SubActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityStack:\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:10.068]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 72/289 failed (KQL: let _startTime = datetime(2026-10-09T04:55:07.0938715Z);\nlet _endTime = datetime(2026-10-09T05:00:07.0938714Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T04:55:07.0938715Z);\nlet _endTime = datetime(2026-10-09T05:00:07.0938714Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:10.102]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-basic-3825519452/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/cosmosContent"
    }�[0m
    �[37m[23:00:10.102]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-basic-3825519452/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:10.310]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:10.310]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-basic-3825519452/cosmosContent"
    }�[0m
    �[37m[23:00:10.320]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:20.724]�[0m �[36mINFO:�[0m �[97mCompleted�[0m �[90m{
      "commits": 294
    }�[0m
    �[37m[23:00:20.724]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect"
    }�[0m
    �[37m[23:00:20.786]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:20.857]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:20.886]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:20.886]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:20.886]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:21.018]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:21.198]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:21.347]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:00:21.884]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:08.6161084Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\", ActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", SubActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityStack:\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\"}, IsPermanent:false})",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452"
    }�[0m
    �[37m[23:00:21.884]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:08.6161084Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\", ActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", SubActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityStack:\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\"}, IsPermanent:false})"
    }�[0m
    
      query-skip-hcp-logs: command failed: exit status 1
    output: �[37m[23:00:22.064]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616"
    }�[0m
    �[37m[23:00:24.931]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:23.5364121Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\", ActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", SubActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:24.931]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 54/289 failed (KQL: let _startTime = datetime(2026-10-09T03:25:22.0635262Z);\nlet _endTime = datetime(2026-10-09T03:30:22.0635261Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:24.965]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/cosmosContent"
    }�[0m
    �[37m[23:00:24.965]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:25.188]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:25.188]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/cosmosContent"
    }�[0m
    �[37m[23:00:25.198]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:31.232]�[0m �[36mINFO:�[0m �[97mCompleted�[0m �[90m{
      "commits": 294
    }�[0m
    �[37m[23:00:31.233]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect"
    }�[0m
    �[37m[23:00:31.296]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:31.363]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:31.391]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:31.391]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:31.391]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:31.520]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:31.665]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:31.785]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:00:32.084]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:23.5364121Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\", ActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", SubActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\"}, IsPermanent:false})",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616"
    }�[0m
    �[37m[23:00:32.084]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:23.5364121Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\", ActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", SubActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\"}, IsPermanent:false})"
    }�[0m
    
      query-skip-kubernetes-events: command failed: exit status 1
    output: �[37m[23:00:32.265]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048"
    }�[0m
    �[37m[23:00:35.196]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:33.7819049Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\", ActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", SubActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityStack:\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:35.196]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 73/289 failed (KQL: let _startTime = datetime(2026-10-09T05:00:32.2650857Z);\nlet _endTime = datetime(2026-10-09T05:05:32.2650856Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:35.196]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/cosmosContent"
    }�[0m
    �[37m[23:00:35.196]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:35.406]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:35.406]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/cosmosContent"
    }�[0m
    �[37m[23:00:35.416]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:41.551]�[0m �[36mINFO:�[0m �[97mCompleted�[0m �[90m{
      "commits": 294
    }�[0m
    �[37m[23:00:41.551]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect"
    }�[0m
    �[37m[23:00:41.611]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:41.684]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:41.712]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:41.713]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:41.713]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:41.839]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:41.983]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:42.104]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:00:42.413]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:33.7819049Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\", ActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", SubActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityStack:\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\"}, IsPermanent:false})",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048"
    }�[0m
    �[37m[23:00:42.413]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:33.7819049Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\", ActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", SubActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityStack:\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\"}, IsPermanent:false})"
    }�[0m
    
      query-collect-systemd-logs: command failed: exit status 1
    output: �[37m[23:00:42.585]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649"
    }�[0m
    �[37m[23:00:46.166]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:44.8927968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\", ActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", SubActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:46.166]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 74/289 failed (KQL: let _startTime = datetime(2026-10-09T05:05:42.5848876Z);\nlet _endTime = datetime(2026-10-09T05:10:42.5848875Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T05:05:42.5848876Z);\nlet _endTime = datetime(2026-10-09T05:10:42.5848875Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:46.229]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/cosmosContent"
    }�[0m
    �[37m[23:00:46.229]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:46.437]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:46.437]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/cosmosContent"
    }�[0m
    �[37m[23:00:46.447]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:49.188]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect"
    }�[0m
    �[37m[23:00:49.244]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:49.310]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:49.336]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:49.336]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:49.336]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:49.455]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:49.591]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:49.701]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:00:49.989]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:44.8927968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\", ActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", SubActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\"}, IsPermanent:false})\nfailed to generate Cosmos content Git history: failed to process entries: failed to stage file: fork/exec /usr/bin/git: resource temporarily unavailable\n",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649"
    }�[0m
    �[37m[23:00:49.989]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:44.8927968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\", ActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", SubActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\"}, IsPermanent:false})\nfailed to generate Cosmos content Git history: failed to process entries: failed to stage file: fork/exec /usr/bin/git: resource temporarily unavailable\n"
    }�[0m
    
      query-skip-hcp-and-events: command failed: exit status 1
    output: �[37m[23:00:50.121]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126"
    }�[0m
    �[37m[23:00:53.109]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:51.6990414Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\", ActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", SubActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityStack:\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:53.110]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 70/289 failed (KQL: let _startTime = datetime(2026-10-09T04:45:50.1214504Z);\nlet _endTime = datetime(2026-10-09T04:50:50.1214503Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T04:45:50.1214504Z);\nlet _endTime = datetime(2026-10-09T04:50:50.1214503Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:53.110]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/cosmosContent"
    }�[0m
    �[37m[23:00:53.110]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:53.326]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:53.326]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/cosmosContent"
    }�[0m
    �[37m[23:00:53.335]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:59.329]�[0m �[36mINFO:�[0m �[97mCompleted�[0m �[90m{
      "commits": 294
    }�[0m
    �[37m[23:00:59.329]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect"
    }�[0m
    �[37m[23:00:59.389]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:59.462]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:59.489]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:59.489]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:59.489]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:59.614]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:59.757]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:59.878]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:01:00.172]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:51.6990414Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\", ActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", SubActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityStack:\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\"}, IsPermanent:false})",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126"
    }�[0m
    �[37m[23:01:00.173]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:51.6990414Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\", ActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", SubActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityStack:\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\"}, IsPermanent:false})"
    }�[0m
    
    {
        s: "must-gather CLI test failures:\n  query-basic: command failed: exit status 1\noutput: \x1b[37m[23:00:07.094]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452\"\n}\x1b[0m\n\x1b[37m[23:00:10.068]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:08.6161084Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\\\", ActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", SubActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \\u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:10.068]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 72/289 failed (KQL: let _startTime = datetime(2026-10-09T04:55:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T05:00:07.0938714Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T04:55:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T05:00:07.0938714Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \\\"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\\\": context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:10.102]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:10.102]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:10.310]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:10.310]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:10.320]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:20.724]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mCompleted\x1b[0m \x1b[90m{\n  \"commits\": 294\n}\x1b[0m\n\x1b[37m[23:00:20.724]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:20.786]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:20.857]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:20.886]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:20.886]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:20.886]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:21.018]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:21.198]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:21.347]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:00:21.884]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:08.6161084Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\\\", ActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", SubActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \\u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\\\"}, IsPermanent:false})\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452\"\n}\x1b[0m\n\x1b[37m[23:00:21.884]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:08.6161084Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\\\", ActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", SubActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \\u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\\\"}, IsPermanent:false})\"\n}\x1b[0m\n\n  query-skip-hcp-logs: command failed: exit status 1\noutput: \x1b[37m[23:00:22.064]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616\"\n}\x1b[0m\n\x1b[37m[23:00:24.931]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:23.5364121Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\\\", ActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", SubActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \\u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:24.931]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 54/289 failed (KQL: let _startTime = datetime(2026-10-09T03:25:22.0635262Z);\\nlet _endTime = datetime(2026-10-09T03:30:22.0635261Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:24.965]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:24.965]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:25.188]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:25.188]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:25.198]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:31.232]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mCompleted\x1b[0m \x1b[90m{\n  \"commits\": 294\n}\x1b[0m\n\x1b[37m[23:00:31.233]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:31.296]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:31.363]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:31.391]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:31.391]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:31.391]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:31.520]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:31.665]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:31.785]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:00:32.084]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:23.5364121Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\\\", ActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", SubActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \\u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\\\"}, IsPermanent:false})\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616\"\n}\x1b[0m\n\x1b[37m[23:00:32.084]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:23.5364121Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\\\", ActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", SubActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \\u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\\\"}, IsPermanent:false})\"\n}\x1b[0m\n\n  query-skip-kubernetes-events: command failed: exit status 1\noutput: \x1b[37m[23:00:32.265]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048\"\n}\x1b[0m\n\x1b[37m[23:00:35.196]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:33.7819049Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\\\", ActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", SubActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \\u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:35.196]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 73/289 failed (KQL: let _startTime = datetime(2026-10-09T05:00:32.2650857Z);\\nlet _endTime = datetime(2026-10-09T05:05:32.2650856Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:35.196]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:35.196]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:35.406]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:35.406]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:35.416]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:41.551]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mCompleted\x1b[0m \x1b[90m{\n  \"commits\": 294\n}\x1b[0m\n\x1b[37m[23:00:41.551]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:41.611]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:41.684]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:41.712]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:41.713]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:41.713]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:41.839]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:41.983]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:42.104]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:00:42.413]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:33.7819049Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\\\", ActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", SubActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \\u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\\\"}, IsPermanent:false})\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048\"\n}\x1b[0m\n\x1b[37m[23:00:42.413]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:33.7819049Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\\\", ActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", SubActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \\u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\\\"}, IsPermanent:false})\"\n}\x1b[0m\n\n  query-collect-systemd-logs: command failed: exit status 1\noutput: \x1b[37m[23:00:42.585]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649\"\n}\x1b[0m\n\x1b[37m[23:00:46.166]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:44.8927968Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\\\", ActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", SubActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \\u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:46.166]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 74/289 failed (KQL: let _startTime = datetime(2026-10-09T05:05:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T05:10:42.5848875Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T05:05:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T05:10:42.5848875Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \\\"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\\\": context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:46.229]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:46.229]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:46.437]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:46.437]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:46.447]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:49.188]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:49.244]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:49.310]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:49.336]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:49.336]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:49.336]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:49.455]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:49.591]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:49.701]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:00:49.989]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:44.8927968Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\\\", ActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", SubActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \\u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\\\"}, IsPermanent:false})\\nfailed to generate Cosmos content Git history: failed to process entries: failed to stage file: fork/exec /usr/bin/git: resource temporarily unavailable\\n\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649\"\n}\x1b[0m\n\x1b[37m[23:00:49.989]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:44.8927968Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\\\", ActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", SubActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \\u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\\\"}, IsPermanent:false})\\nfailed to generate Cosmos content Git history: failed to process entries: failed to stage file: fork/exec /usr/bin/git: resource temporarily unavailable\\n\"\n}\x1b[0m\n\n  query-skip-hcp-and-events: command failed: exit status 1\noutput: \x1b[37m[23:00:50.121]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126\"\n}\x1b[0m\n\x1b[37m[23:00:53.109]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:51.6990414Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\\\", ActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", SubActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \\u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:53.110]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 70/289 failed (KQL: let _startTime = datetime(2026-10-09T04:45:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T04:50:50.1214503Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T04:45:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T04:50:50.1214503Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \\\"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\\\": context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:53.110]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:53.110]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:53.326]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:53.326]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:53.335]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:59.329]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mCompleted\x1b[0m \x1b[90m{\n  \"commits\": 294\n}\x1b[0m\n\x1b[37m[23:00:59.329]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:59.389]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:59.462]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:59.489]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:59.489]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:59.489]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:59.614]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:59.757]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:59.878]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:01:00.172]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:51.6990414Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\\\", ActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", SubActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \\u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\\\"}, IsPermanent:false})\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126\"\n}\x1b[0m\n\x1b[37m[23:01:00.173]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:51.6990414Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\\\", ActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", SubActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \\u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\\\"}, IsPermanent:false})\"\n}\x1b[0m\n",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 651.764s.
Expected success, but got an error:
    <*errors.errorString | 0x104322d955a0>: 
    must-gather CLI test failures:
      query-basic: command failed: exit status 1
    output: �[37m[23:00:07.094]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452"
    }�[0m
    �[37m[23:00:10.068]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:08.6161084Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\", ActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", SubActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityStack:\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:10.068]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 72/289 failed (KQL: let _startTime = datetime(2026-10-09T04:55:07.0938715Z);\nlet _endTime = datetime(2026-10-09T05:00:07.0938714Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T04:55:07.0938715Z);\nlet _endTime = datetime(2026-10-09T05:00:07.0938714Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:10.102]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-basic-3825519452/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/cosmosContent"
    }�[0m
    �[37m[23:00:10.102]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-basic-3825519452/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:10.310]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:10.310]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-basic-3825519452/cosmosContent"
    }�[0m
    �[37m[23:00:10.320]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:20.724]�[0m �[36mINFO:�[0m �[97mCompleted�[0m �[90m{
      "commits": 294
    }�[0m
    �[37m[23:00:20.724]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect"
    }�[0m
    �[37m[23:00:20.786]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:20.857]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:20.886]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:20.886]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:20.886]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:21.018]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:21.198]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:21.347]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:00:21.884]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:08.6161084Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\", ActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", SubActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityStack:\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\"}, IsPermanent:false})",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452"
    }�[0m
    �[37m[23:00:21.884]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:08.6161084Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\", ActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", SubActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityStack:\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\"}, IsPermanent:false})"
    }�[0m
    
      query-skip-hcp-logs: command failed: exit status 1
    output: �[37m[23:00:22.064]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616"
    }�[0m
    �[37m[23:00:24.931]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:23.5364121Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\", ActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", SubActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:24.931]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 54/289 failed (KQL: let _startTime = datetime(2026-10-09T03:25:22.0635262Z);\nlet _endTime = datetime(2026-10-09T03:30:22.0635261Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:24.965]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/cosmosContent"
    }�[0m
    �[37m[23:00:24.965]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:25.188]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:25.188]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/cosmosContent"
    }�[0m
    �[37m[23:00:25.198]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:31.232]�[0m �[36mINFO:�[0m �[97mCompleted�[0m �[90m{
      "commits": 294
    }�[0m
    �[37m[23:00:31.233]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect"
    }�[0m
    �[37m[23:00:31.296]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:31.363]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:31.391]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:31.391]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:31.391]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:31.520]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:31.665]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:31.785]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:00:32.084]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:23.5364121Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\", ActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", SubActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\"}, IsPermanent:false})",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616"
    }�[0m
    �[37m[23:00:32.084]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:23.5364121Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\", ActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", SubActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\"}, IsPermanent:false})"
    }�[0m
    
      query-skip-kubernetes-events: command failed: exit status 1
    output: �[37m[23:00:32.265]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048"
    }�[0m
    �[37m[23:00:35.196]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:33.7819049Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\", ActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", SubActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityStack:\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:35.196]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 73/289 failed (KQL: let _startTime = datetime(2026-10-09T05:00:32.2650857Z);\nlet _endTime = datetime(2026-10-09T05:05:32.2650856Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:35.196]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/cosmosContent"
    }�[0m
    �[37m[23:00:35.196]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:35.406]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:35.406]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/cosmosContent"
    }�[0m
    �[37m[23:00:35.416]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:41.551]�[0m �[36mINFO:�[0m �[97mCompleted�[0m �[90m{
      "commits": 294
    }�[0m
    �[37m[23:00:41.551]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect"
    }�[0m
    �[37m[23:00:41.611]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:41.684]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:41.712]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:41.713]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:41.713]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:41.839]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:41.983]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:42.104]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:00:42.413]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:33.7819049Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\", ActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", SubActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityStack:\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\"}, IsPermanent:false})",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048"
    }�[0m
    �[37m[23:00:42.413]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:33.7819049Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\", ActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", SubActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityStack:\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\"}, IsPermanent:false})"
    }�[0m
    
      query-collect-systemd-logs: command failed: exit status 1
    output: �[37m[23:00:42.585]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649"
    }�[0m
    �[37m[23:00:46.166]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:44.8927968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\", ActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", SubActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:46.166]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 74/289 failed (KQL: let _startTime = datetime(2026-10-09T05:05:42.5848876Z);\nlet _endTime = datetime(2026-10-09T05:10:42.5848875Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T05:05:42.5848876Z);\nlet _endTime = datetime(2026-10-09T05:10:42.5848875Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:46.229]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/cosmosContent"
    }�[0m
    �[37m[23:00:46.229]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:46.437]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:46.437]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/cosmosContent"
    }�[0m
    �[37m[23:00:46.447]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:49.188]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect"
    }�[0m
    �[37m[23:00:49.244]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:49.310]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:49.336]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:49.336]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:49.336]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:49.455]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:49.591]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:49.701]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:00:49.989]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:44.8927968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\", ActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", SubActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\"}, IsPermanent:false})\nfailed to generate Cosmos content Git history: failed to process entries: failed to stage file: fork/exec /usr/bin/git: resource temporarily unavailable\n",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649"
    }�[0m
    �[37m[23:00:49.989]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:44.8927968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\", ActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", SubActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\"}, IsPermanent:false})\nfailed to generate Cosmos content Git history: failed to process entries: failed to stage file: fork/exec /usr/bin/git: resource temporarily unavailable\n"
    }�[0m
    
      query-skip-hcp-and-events: command failed: exit status 1
    output: �[37m[23:00:50.121]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126"
    }�[0m
    �[37m[23:00:53.109]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:51.6990414Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\", ActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", SubActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityStack:\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:53.110]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 70/289 failed (KQL: let _startTime = datetime(2026-10-09T04:45:50.1214504Z);\nlet _endTime = datetime(2026-10-09T04:50:50.1214503Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T04:45:50.1214504Z);\nlet _endTime = datetime(2026-10-09T04:50:50.1214503Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:53.110]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/cosmosContent"
    }�[0m
    �[37m[23:00:53.110]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:53.326]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:53.326]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/cosmosContent"
    }�[0m
    �[37m[23:00:53.335]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:59.329]�[0m �[36mINFO:�[0m �[97mCompleted�[0m �[90m{
      "commits": 294
    }�[0m
    �[37m[23:00:59.329]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect"
    }�[0m
    �[37m[23:00:59.389]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:59.462]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:59.489]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:59.489]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:59.489]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:59.614]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:59.757]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:59.878]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:01:00.172]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:51.6990414Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\", ActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", SubActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityStack:\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\"}, IsPermanent:false})",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126"
    }�[0m
    �[37m[23:01:00.173]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:51.6990414Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\", ActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", SubActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityStack:\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\"}, IsPermanent:false})"
    }�[0m
    
    {
        s: "must-gather CLI test failures:\n  query-basic: command failed: exit status 1\noutput: \x1b[37m[23:00:07.094]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452\"\n}\x1b[0m\n\x1b[37m[23:00:10.068]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:08.6161084Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\\\", ActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", SubActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \\u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:10.068]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 72/289 failed (KQL: let _startTime = datetime(2026-10-09T04:55:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T05:00:07.0938714Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T04:55:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T05:00:07.0938714Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \\\"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\\\": context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:10.102]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:10.102]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:10.310]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:10.310]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:10.320]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:20.724]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mCompleted\x1b[0m \x1b[90m{\n  \"commits\": 294\n}\x1b[0m\n\x1b[37m[23:00:20.724]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:20.786]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:20.857]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:20.886]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:20.886]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:20.886]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:21.018]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:21.198]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:21.347]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:00:21.884]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:08.6161084Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\\\", ActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", SubActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \\u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\\\"}, IsPermanent:false})\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452\"\n}\x1b[0m\n\x1b[37m[23:00:21.884]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:08.6161084Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\\\", ActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", SubActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \\u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\\\"}, IsPermanent:false})\"\n}\x1b[0m\n\n  query-skip-hcp-logs: command failed: exit status 1\noutput: \x1b[37m[23:00:22.064]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616\"\n}\x1b[0m\n\x1b[37m[23:00:24.931]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:23.5364121Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\\\", ActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", SubActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \\u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:24.931]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 54/289 failed (KQL: let _startTime = datetime(2026-10-09T03:25:22.0635262Z);\\nlet _endTime = datetime(2026-10-09T03:30:22.0635261Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:24.965]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:24.965]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:25.188]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:25.188]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:25.198]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:31.232]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mCompleted\x1b[0m \x1b[90m{\n  \"commits\": 294\n}\x1b[0m\n\x1b[37m[23:00:31.233]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:31.296]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:31.363]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:31.391]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:31.391]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:31.391]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:31.520]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:31.665]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:31.785]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:00:32.084]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:23.5364121Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\\\", ActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", SubActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \\u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\\\"}, IsPermanent:false})\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616\"\n}\x1b[0m\n\x1b[37m[23:00:32.084]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:23.5364121Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\\\", ActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", SubActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \\u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\\\"}, IsPermanent:false})\"\n}\x1b[0m\n\n  query-skip-kubernetes-events: command failed: exit status 1\noutput: \x1b[37m[23:00:32.265]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048\"\n}\x1b[0m\n\x1b[37m[23:00:35.196]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:33.7819049Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\\\", ActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", SubActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \\u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:35.196]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 73/289 failed (KQL: let _startTime = datetime(2026-10-09T05:00:32.2650857Z);\\nlet _endTime = datetime(2026-10-09T05:05:32.2650856Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:35.196]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:35.196]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:35.406]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:35.406]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:35.416]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:41.551]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mCompleted\x1b[0m \x1b[90m{\n  \"commits\": 294\n}\x1b[0m\n\x1b[37m[23:00:41.551]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:41.611]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:41.684]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:41.712]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:41.713]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:41.713]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:41.839]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:41.983]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:42.104]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:00:42.413]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:33.7819049Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\\\", ActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", SubActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \\u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\\\"}, IsPermanent:false})\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048\"\n}\x1b[0m\n\x1b[37m[23:00:42.413]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:33.7819049Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\\\", ActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", SubActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \\u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\\\"}, IsPermanent:false})\"\n}\x1b[0m\n\n  query-collect-systemd-logs: command failed: exit status 1\noutput: \x1b[37m[23:00:42.585]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649\"\n}\x1b[0m\n\x1b[37m[23:00:46.166]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:44.8927968Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\\\", ActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", SubActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \\u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:46.166]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 74/289 failed (KQL: let _startTime = datetime(2026-10-09T05:05:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T05:10:42.5848875Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T05:05:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T05:10:42.5848875Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \\\"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\\\": context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:46.229]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:46.229]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:46.437]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:46.437]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:46.447]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:49.188]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:49.244]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:49.310]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:49.336]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:49.336]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:49.336]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:49.455]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:49.591]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:49.701]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:00:49.989]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:44.8927968Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\\\", ActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", SubActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \\u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\\\"}, IsPermanent:false})\\nfailed to generate Cosmos content Git history: failed to process entries: failed to stage file: fork/exec /usr/bin/git: resource temporarily unavailable\\n\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649\"\n}\x1b[0m\n\x1b[37m[23:00:49.989]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:44.8927968Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\\\", ActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", SubActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \\u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\\\"}, IsPermanent:false})\\nfailed to generate Cosmos content Git history: failed to process entries: failed to stage file: fork/exec /usr/bin/git: resource temporarily unavailable\\n\"\n}\x1b[0m\n\n  query-skip-hcp-and-events: command failed: exit status 1\noutput: \x1b[37m[23:00:50.121]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126\"\n}\x1b[0m\n\x1b[37m[23:00:53.109]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:51.6990414Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\\\", ActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", SubActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \\u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:53.110]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 70/289 failed (KQL: let _startTime = datetime(2026-10-09T04:45:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T04:50:50.1214503Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T04:45:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T04:50:50.1214503Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \\\"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\\\": context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:53.110]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:53.110]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:53.326]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:53.326]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:53.335]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:59.329]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mCompleted\x1b[0m \x1b[90m{\n  \"commits\": 294\n}\x1b[0m\n\x1b[37m[23:00:59.329]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:59.389]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:59.462]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:59.489]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:59.489]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:59.489]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:59.614]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:59.757]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:59.878]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:01:00.172]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:51.6990414Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\\\", ActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", SubActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \\u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\\\"}, IsPermanent:false})\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126\"\n}\x1b[0m\n\x1b[37m[23:01:00.173]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:51.6990414Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\\\", ActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", SubActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \\u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\\\"}, IsPermanent:false})\"\n}\x1b[0m\n",
    }
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelEngineering should be able to retrieve kusto logs for a cluster and services1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74462108673315018117120
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod networks, namespaces; [descendantResourc...
full failure pattern:
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior history
e2e11.69%1 of 59 job runs affectedIndeterminateSignal: Indeterminate — no prior history; active 3 day(s) — New failure pattern — no prior historyShow trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 1 · Oct 7: 0 · Oct 8: 1 · Oct 9: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2403780381b0>: 
    resource group "rg-candidate-4-22-2twmlnh2pntd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-97w8j7" in resourcegroup="rg-candidate-4-22-2twmlnh2pntd" to finish deleting: GET https://rp.j2706688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec134b00-506f-4598-8f43-9039b6949cbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec134b00-506f-4598-8f43-9039b6949cbf",
      "name": "ec134b00-506f-4598-8f43-9039b6949cbf",
      "status": "Failed",
      "startTime": "2026-10-09T12:19:56.634862294Z",
      "endTime": "2026-10-09T12:44:04.410412622Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2403780381b0>: 
    resource group "rg-candidate-4-22-2twmlnh2pntd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-97w8j7" in resourcegroup="rg-candidate-4-22-2twmlnh2pntd" to finish deleting: GET https://rp.j2706688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec134b00-506f-4598-8f43-9039b6949cbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec134b00-506f-4598-8f43-9039b6949cbf",
      "name": "ec134b00-506f-4598-8f43-9039b6949cbf",
      "status": "Failed",
      "startTime": "2026-10-09T12:19:56.634862294Z",
      "endTime": "2026-10-09T12:44:04.410412622Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.221
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74372108513309102706688
alert [svc] userJourneyClusterProvisionErrors1h5m fired
full failure pattern:
alert [svc] userJourneyClusterProvisionErrors1h5m fired
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
alert11.69%1 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 1none
Full failure examples (1)
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T18:38:22Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterProvisionErrors1h5m", cluster="ci01-j1936896-svc", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1936896/providers/microsoft.monitor/accounts/services-j1936896", region="westus3", severity="3", short_window="5m", slo="cluster-provision-errors"
  Description: More than 72% of completed cluster create (install) operations on ci01-j1936896-svc failed over the last hour with at least 3 failures, a 14.4x burn of the 95% SLO budget.
Contributing tests (1)
Failed AtJobTestSupport
alertpull-ci-Azure-ARO-HCP-main-e2e-parallel[aro-hcp-observability] [svc] alert userJourneyClusterProvisionErrors1h5m does not fire1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #74402108607415241936896
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
full failure pattern:
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
Signal: IndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)
e2e11.69%1 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 3 prior week(s); active 1 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 1none
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x19485d7e23a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-h78g452l2k2m" to finish creating: GET https://rp.j1850240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2445641e-62af-46ed-8294-3ebf3cab568c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2445641e-62af-46ed-8294-3ebf3cab568c",
      "name": "2445641e-62af-46ed-8294-3ebf3cab568c",
      "status": "Failed",
      "startTime": "2026-10-09T03:36:09.491861222Z",
      "endTime": "2026-10-09T04:05:13.503264505Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-h78g452l2k2m--managed/providers/Microsoft.Compute/virtualMachines/t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-h78g452l2k2m--managed/providers/Microsoft.Compute/virtualMachines/t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x19485d7e23a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-h78g452l2k2m" to finish creating: GET https://rp.j1850240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2445641e-62af-46ed-8294-3ebf3cab568c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2445641e-62af-46ed-8294-3ebf3cab568c",
      "name": "2445641e-62af-46ed-8294-3ebf3cab568c",
      "status": "Failed",
      "startTime": "2026-10-09T03:36:09.491861222Z",
      "endTime": "2026-10-09T04:05:13.503264505Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-h78g452l2k2m--managed/providers/Microsoft.Compute/virtualMachines/t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-h78g452l2k2m--managed/providers/Microsoft.Compute/virtualMachines/t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
Contributing tests (1)
Failed AtJobTestSupport
e2epull-ci-Azure-ARO-HCP-main-e2e-parallelCustomer should be able to delete a nodepool1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #72922108377922371850240
failed to roll out Helm release: resource DaemonSet/swift-recorder/swift-recorder not ready. status: InProgress, message: Available: 16/17 context deadline exceeded
full failure pattern:
failed to roll out Helm release: resource DaemonSet/swift-recorder/swift-recorder not ready. status: InProgress, message: Available: 16/17 context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)
provision11.69%1 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 1 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 1none
Full failure examples (1)
time=2026-10-09T15:06:37.853Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 description="Step deploy\n Kind: Helm\n"
time=2026-10-09T15:06:38.448Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 values="map[captureMode:all clusterName:ci01-j4897024-mgmt-2 enabled:true environment:ci01 image:map[digest:sha256:11b789c31d680e7f988ca6ecb8901408801a582f560ea82572d969f5ddb243a6 registry:arohcpsvcdev.azurecr.io repository:ci-op-crcb6znc/pipeline] region:westus3 resources:map[limits:map[memory:256Mi] requests:map[cpu:20m memory:64Mi]] serviceAccount:map[name:swift-recorder]]"
time=2026-10-09T15:06:38.449Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2
time=2026-10-09T15:06:38.710Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 namespace=swift-recorder
time=2026-10-09T15:06:39.018Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 dryRun=false
time=2026-10-09T15:06:39.018Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2
time=2026-10-09T15:06:39.140Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2
time=2026-10-09T15:16:42.762Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 err="resource DaemonSet/swift-recorder/swift-recorder not ready. status: InProgress, message: Available: 16/17\ncontext deadline exceeded"
time=2026-10-09T15:16:42.762Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/swift-recorder/swift-recorder not ready. status: InProgress, message: Available: 16/17\ncontext deadline exceeded"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.SwiftRecorder/management/deploy (stamp=2)1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73972108558761344897024
failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded
full failure pattern:
failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded
Signal: IndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)
provision11.69%1 of 59 job runs affectedIndeterminateSignal: Indeterminate — seen in 2 prior week(s); active 1 day(s)Show trend detailsOct 3: 0 · Oct 4: 0 · Oct 5: 0 · Oct 6: 0 · Oct 7: 0 · Oct 8: 0 · Oct 9: 1none
Full failure examples (1)
time=2026-10-09T03:02:25.127Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-09T03:02:27.026Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-09T03:02:29.104Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=9100c383496433b8981e04307d4ff94d8b8595203688da3bb283a6fce4e49fea portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9299456-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9100c383496433b8981e04307d4ff94d8b8595203688da3bb283a6fce4e49fea
time=2026-10-09T03:32:24.844Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"
Contributing tests (1)
Failed AtJobTestSupport
provisionpull-ci-Azure-ARO-HCP-main-e2e-parallelRun pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster1
Affected runs (1)
Date (UTC)Associated PRProw job
PR #73942108385613169299456

Environment: PROD

Failure patterns: 4 · Failures matched: 27 · Total runs: 9 · Runs affected: 2

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 7 days
e2e111.11%1 of 9 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 7 daysShow trend detailsOct 3: 2 · Oct 4: 2 · Oct 5: 2 · Oct 6: 2 · Oct 7: 2 · Oct 8: 5 · Oct 9: 5STG
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x2a25db7081b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-gkw6wzhpx84w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x2a25db7081b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-gkw6wzhpx84w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x30713b85be60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-tffs96fmr5p6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x30713b85be60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-tffs96fmr5p6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x284ab7439230>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-4qsn8gb8p28c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x284ab7439230>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-4qsn8gb8p28c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2108694214656135168
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenSh...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)
e2e111.11%1 of 9 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)Show trend detailsOct 3: 15 · Oct 4: 16 · Oct 5: 17 · Oct 6: 18 · Oct 7: 17 · Oct 8: 10 · Oct 9: 10none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-s865cq" to 4.21
Unexpected error:
    <*exported.ResponseError | 0x29e047bb9ce0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-84fl6x5gkgjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-s865cq
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-s865cq" to 4.21
Unexpected error:
    <*exported.ResponseError | 0x29e047bb9ce0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-84fl6x5gkgjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-s865cq
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-pwmrlt" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x364694577140>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-bcxf7z6jvkdm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-pwmrlt
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-pwmrlt" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x364694577140>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-bcxf7z6jvkdm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-pwmrlt
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-qldg5l" to 5.0
Unexpected error:
    <*exported.ResponseError | 0x200557060510>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-hldpsj9c8292/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-qldg5l
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-qldg5l" to 5.0
Unexpected error:
    <*exported.ResponseError | 0x200557060510>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-hldpsj9c8292/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-qldg5l
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (10)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.211
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2108377065311965184
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenS...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.nodepool.max-update-duration": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)
e2e111.11%1 of 9 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)Show trend detailsOct 3: 11 · Oct 4: 11 · Oct 5: 12 · Oct 6: 12 · Oct 7: 12 · Oct 8: 9 · Oct 9: 9none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x6f0cf085080>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-6zf46c" resourcegroup="rg-np-dg-minor-6zf46c-hll6hfr44s9d": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-6zf46c-hll6hfr44s9d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-6zf46c/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x6f0cf085080>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-6zf46c" resourcegroup="rg-np-dg-minor-6zf46c-hll6hfr44s9d": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-6zf46c-hll6hfr44s9d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-6zf46c/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x1508ab4d7940>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-lc4jfj" resourcegroup="rg-np-version-upgrade-lc4jfj-qb5cmdpk8ppz": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lc4jfj-qb5cmdpk8ppz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-lc4jfj/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x1508ab4d7940>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-lc4jfj" resourcegroup="rg-np-version-upgrade-lc4jfj-qb5cmdpk8ppz": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lc4jfj-qb5cmdpk8ppz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-lc4jfj/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.41 to 4.22.17
Unexpected error:
    <*fmt.wrapError | 0x22aa54096dc0>: 
    failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-qs6slk" resourcegroup="rg-np-skip-minor-qs6slk-svmgzjwq29hn": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-qs6slk-svmgzjwq29hn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-qs6slk/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.41 to 4.22.17
Unexpected error:
    <*fmt.wrapError | 0x22aa54096dc0>: 
    failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-qs6slk" resourcegroup="rg-np-skip-minor-qs6slk-svmgzjwq29hn": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-qs6slk-svmgzjwq29hn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-qs6slk/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (9)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should be able to update node pool labels and taints1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.201
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelCustomer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2108377065311965184
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpe...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "aro-hcp.experimental.cluster.z-stream-update-policy": unrecognized experimental tag; provider Microsoft.RedHatOpenShift
Signal: IndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)
e2e111.11%1 of 9 job runs affectedIndeterminateSignal: Indeterminate — seen in 1 prior week(s); active 7 day(s)Show trend detailsOct 3: 3 · Oct 4: 2 · Oct 5: 3 · Oct 6: 2 · Oct 7: 4 · Oct 8: 3 · Oct 9: 3none
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-g5xfmb" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x2d844c21d830>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-5xdvnklmwfq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-g5xfmb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-g5xfmb" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x2d844c21d830>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-5xdvnklmwfq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-g5xfmb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-t4k2nv" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x9a9211d2ff0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-wdg9f8phhv6h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-t4k2nv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-t4k2nv" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x9a9211d2ff0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-wdg9f8phhv6h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-t4k2nv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-kqzff9" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xe01aaa2bbc0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7pmn5ng5wrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-kqzff9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-kqzff9" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xe01aaa2bbc0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7pmn5ng5wrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-kqzff9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (3)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.221
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.231
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallelService Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.01
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2108377065311965184

Environment: STG

Failure patterns: 2 · Failures matched: 6 · Total runs: 4 · Runs affected: 2

Failure patternThe canonical description of a recurring CI failure, extracted and normalized from raw logs.Failed atThe stage of the job run where this failure occurred: 'provision' (environment setup, DEV only), 'e2e' (test suite execution), or 'other' (CI infrastructure issues that did not produce a failure pattern).Number of distinct job runs where this failure pattern was detected.Percentage of all job runs in this environment affected by this failure pattern during the selected window.Classification of this failure pattern: Regression (likely caused by a specific PR), Flake (intermittent failure spread across days), Noise (low-quality or generic pattern), or Indeterminate.TrendShows daily activity for this failure pattern in a trailing window anchored to the selected end date. The sparkline covers at least 7 days and at most 14 days, depending on the current window size.Also inOther environments where the same failure pattern was also detected during the selected window.
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when c...
full failure pattern:
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
Signal: FlakeSignal: Flake — present in 3 prior week(s); spread across 6 days
e2e125.00%1 of 4 job runs affectedFlakeSignal: Flake — present in 3 prior week(s); spread across 6 daysShow trend detailsOct 3: 5 · Oct 4: 0 · Oct 5: 5 · Oct 6: 5 · Oct 7: 5 · Oct 8: 5 · Oct 9: 5PROD
Full failure examples (3)
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x16ef84387050>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-8hwhxx82br9m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x16ef84387050>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-8hwhxx82br9m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x3fc6670d91a0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-mgmsfgq99bg9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x3fc6670d91a0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-mgmsfgq99bg9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x3f9b94218c90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-jr65q449l4wp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x3f9b94218c90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-jr65q449l4wp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (5)
Failed AtJobTestSupport
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyAuthorized CIDRs Connectivity should allow API access only from authorized VM IP1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create a cluster and node pools with aggregated advanced features1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster and manage ImageDigestMirrors1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyCustomer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration1
e2eperiodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightlyUpdate HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2108694214664523776
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
full failure pattern:
ERROR CODE: DeadlineExceeded; provider Microsoft.Compute
Signal: FlakeSignal: Flake — present in 2 prior week(s); spread across 7 days
e2e125.00%1 of 4 job runs affectedFlakeSignal: Flake — present in 2 prior week(s); spread across 7 daysShow trend detailsOct 3: 2 · Oct 4: 1 · Oct 5: 1 · Oct 6: 18 · Oct 7: 40 · Oct 8: 9 · Oct 9: 1DEV
Full failure examples (1)
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x13cc8f58b420>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-sz5gdnfzqhdd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c72bb6d-878b-457e-bf39-56f98851abda
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c72bb6d-878b-457e-bf39-56f98851abda",
      "name": "2c72bb6d-878b-457e-bf39-56f98851abda",
      "status": "Failed",
      "startTime": "2026-10-09T06:39:07.400427011Z",
      "endTime": "2026-10-09T07:08:09.774067703Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-sz5gdnfzqhdd--managed/providers/Microsoft.Compute/virtualMachines/y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-sz5gdnfzqhdd--managed/providers/Microsoft.Compute/virtualMachines/y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x13cc8f58b420>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-sz5gdnfzqhdd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c72bb6d-878b-457e-bf39-56f98851abda
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c72bb6d-878b-457e-bf39-56f98851abda",
      "name": "2c72bb6d-878b-457e-bf39-56f98851abda",
      "status": "Failed",
      "startTime": "2026-10-09T06:39:07.400427011Z",
      "endTime": "2026-10-09T07:08:09.774067703Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-sz5gdnfzqhdd--managed/providers/Microsoft.Compute/virtualMachines/y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-sz5gdnfzqhdd--managed/providers/Microsoft.Compute/virtualMachines/y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
Contributing tests (1)
Failed AtJobTestSupport
e2ebranch-ci-Azure-ARO-HCP-main-e2e-stage-e2e-parallelCustomer should be able to cordon, drain, and uncordon a node in an HCP cluster1
Affected runs (1)
Date (UTC)Associated PRProw job
n/a2108427018348531712