alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-10T00:27:09Z
  Ended:       2026-10-10T00:33:08Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8064256-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbj1c9chavhk90qkot47r4fc981jv3j-p7l5u8c0n3u2s1c", pod="router-5869c468cd-cr9j5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbj1c9chavhk90qkot47r4fc981jv3j-p7l5u8c0n3u2s1c/router-5869c468cd-cr9j5 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-10T00:52:27Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j9187456-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.0:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9187456/providers/microsoft.monitor/accounts/services-j9187456", namespace="klusterlet-2tbj2sv7sdukmvi25teg7m48qmbbnkqf", pod="arohcp-monitor-kube-state-metrics-7988dc648c-ns4ml", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tbj2sv7sdukmvi25teg7m48qmbbnkqf on cluster ci01-j9187456-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-10T01:13:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="cloud-network-config-controller-7847666f8b-xtgjl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="78775d9a-b0ca-4c4d-bd49-d00b20e49e68"
  Description: Pod ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/cloud-network-config-controller-7847666f8b-xtgjl (controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-10T01:13:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="csi-snapshot-controller-669f7dfb7b-68ftn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6585df9e-c94d-48b0-a688-96a3851ac4dd"
  Description: Pod ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/csi-snapshot-controller-669f7dfb7b-68ftn (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-10T01:14:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="csi-snapshot-controller-669f7dfb7b-68ftn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="6585df9e-c94d-48b0-a688-96a3851ac4dd"
  Description: Pod ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/csi-snapshot-controller-669f7dfb7b-68ftn (snapshot-controller) is in waiting state (reason: "CrashLoopBackOff").
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-10T01:14:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="cloud-network-config-controller-7847666f8b-xtgjl", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="78775d9a-b0ca-4c4d-bd49-d00b20e49e68"
  Description: Pod ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/cloud-network-config-controller-7847666f8b-xtgjl (controller) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-10T00:42:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mchpr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-10T00:43:38Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j5177088-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.148:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5177088/providers/microsoft.monitor/accounts/hcps-j5177088", namespace="ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mchpr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbitkkae9upoufmp2r1juh69i34b35n-y0t1t6j2p7i5j5k/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3475813fe160>: 
    DNS for route host agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x3475813fe0e0>{
            msg: "DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x3475813fe160>: 
    DNS for route host agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x3475813fe0e0>{
            msg: "DNS for agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-p6pqz.apps.aro.v20261001.e223.j7272832.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

time=2026-10-09T22:05:07.216Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-09T22:06:09.182Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-09T22:05:07.225651655Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7960192-mgmt-2\",\"clusterName\":\"ci01-j7960192-mgmt-2\",\"region\":\"westus3\"}\n{\"time\":\"2026-10-09T22:05:08.960792766Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7960192-mgmt-2\",\"clusterName\":\"ci01-j7960192-mgmt-2\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-09T22:05:09.292773856Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7960192-mgmt-2\",\"clusterName\":\"ci01-j7960192-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-10-09T22:06:09.178268762Z\",\"level\":\"ERROR\",\"msg\":\"collecting deployment operations\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7960192-mgmt-2\",\"clusterName\":\"ci01-j7960192-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"err\":{\"errorCode\":\"DeploymentNotFound\",\"statusCode\":404,\"errorMessage\":\"GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7960192-mgmt-2/deployments/aks-cluster-create/operations\\n--------------------------------------------------------------------------------\\nRESPONSE 404: 404 Not Found\\nERROR CODE: DeploymentNotFound\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"DeploymentNotFound\\\",\\n    \\\"message\\\": \\\"Deployment 'aks-cluster-create' could not be found.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"}}\n{\"time\":\"2026-10-09T22:06:09.178359704Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j7960192-mgmt-2\",\"clusterName\":\"ci01-j7960192-mgmt-2\",\"region\":\"westus3\",\"err\":\"submitting deployment aks-cluster-create: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7960192-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\n--------------------------------------------------------------------------------\\nRESPONSE 400: 400 Bad Request\\nERROR CODE: InvalidTemplateDeployment\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"InvalidTemplateDeployment\\\",\\n    \\\"message\\\": \\\"The template deployment 'aks-cluster-create' is not valid according to the validation procedure. The following resource provider(s) - 'Microsoft.ContainerService/managedClusters (2026-04-02-preview)' reported preflight validation errors. Tracking id is 'af747446-bfb7-43e9-a875-3f8953ec8e67'. See inner errors for details.\\\",\\n    \\\"details\\\": [\\n      {\\n        \\\"code\\\": \\\"ListResourceSKUsFailed\\\",\\n        \\\"message\\\": \\\"Preflight validation check for resource(s) for container service ci01-j7960192-mgmt-2 in resource group hcp-underlay-ci01-j7960192-mgmt-2 failed. Message: List resource SKUs in subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX failed. Details: \\\"\\n      }\\n    ]\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"1m1.952725139s\"}\n exit status 1"

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 651.764s.
Expected success, but got an error:
    <*errors.errorString | 0x104322d955a0>: 
    must-gather CLI test failures:
      query-basic: command failed: exit status 1
    output: �[37m[23:00:07.094]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452"
    }�[0m
    �[37m[23:00:10.068]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:08.6161084Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\", ActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", SubActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityStack:\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:10.068]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 72/289 failed (KQL: let _startTime = datetime(2026-10-09T04:55:07.0938715Z);\nlet _endTime = datetime(2026-10-09T05:00:07.0938714Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T04:55:07.0938715Z);\nlet _endTime = datetime(2026-10-09T05:00:07.0938714Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:10.102]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-basic-3825519452/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/cosmosContent"
    }�[0m
    �[37m[23:00:10.102]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-basic-3825519452/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:10.310]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:10.310]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-basic-3825519452/cosmosContent"
    }�[0m
    �[37m[23:00:10.320]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:20.724]�[0m �[36mINFO:�[0m �[97mCompleted�[0m �[90m{
      "commits": 294
    }�[0m
    �[37m[23:00:20.724]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect"
    }�[0m
    �[37m[23:00:20.786]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:20.857]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:20.886]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:20.886]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:20.886]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:21.018]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:21.198]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:21.347]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:00:21.884]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:08.6161084Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\", ActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", SubActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityStack:\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\"}, IsPermanent:false})",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452"
    }�[0m
    �[37m[23:00:21.884]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:08.6161084Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\", ActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", SubActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityStack:\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\"}, IsPermanent:false})"
    }�[0m
    
      query-skip-hcp-logs: command failed: exit status 1
    output: �[37m[23:00:22.064]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616"
    }�[0m
    �[37m[23:00:24.931]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:23.5364121Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\", ActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", SubActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:24.931]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 54/289 failed (KQL: let _startTime = datetime(2026-10-09T03:25:22.0635262Z);\nlet _endTime = datetime(2026-10-09T03:30:22.0635261Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:24.965]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/cosmosContent"
    }�[0m
    �[37m[23:00:24.965]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:25.188]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:25.188]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/cosmosContent"
    }�[0m
    �[37m[23:00:25.198]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:31.232]�[0m �[36mINFO:�[0m �[97mCompleted�[0m �[90m{
      "commits": 294
    }�[0m
    �[37m[23:00:31.233]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect"
    }�[0m
    �[37m[23:00:31.296]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:31.363]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:31.391]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:31.391]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:31.391]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:31.520]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:31.665]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:31.785]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:00:32.084]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:23.5364121Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\", ActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", SubActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\"}, IsPermanent:false})",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616"
    }�[0m
    �[37m[23:00:32.084]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:23.5364121Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\", ActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", SubActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\"}, IsPermanent:false})"
    }�[0m
    
      query-skip-kubernetes-events: command failed: exit status 1
    output: �[37m[23:00:32.265]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048"
    }�[0m
    �[37m[23:00:35.196]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:33.7819049Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\", ActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", SubActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityStack:\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:35.196]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 73/289 failed (KQL: let _startTime = datetime(2026-10-09T05:00:32.2650857Z);\nlet _endTime = datetime(2026-10-09T05:05:32.2650856Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:35.196]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/cosmosContent"
    }�[0m
    �[37m[23:00:35.196]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:35.406]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:35.406]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/cosmosContent"
    }�[0m
    �[37m[23:00:35.416]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:41.551]�[0m �[36mINFO:�[0m �[97mCompleted�[0m �[90m{
      "commits": 294
    }�[0m
    �[37m[23:00:41.551]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect"
    }�[0m
    �[37m[23:00:41.611]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:41.684]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:41.712]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:41.713]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:41.713]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:41.839]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:41.983]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:42.104]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:00:42.413]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:33.7819049Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\", ActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", SubActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityStack:\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\"}, IsPermanent:false})",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048"
    }�[0m
    �[37m[23:00:42.413]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:33.7819049Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\", ActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", SubActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityStack:\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\"}, IsPermanent:false})"
    }�[0m
    
      query-collect-systemd-logs: command failed: exit status 1
    output: �[37m[23:00:42.585]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649"
    }�[0m
    �[37m[23:00:46.166]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:44.8927968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\", ActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", SubActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:46.166]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 74/289 failed (KQL: let _startTime = datetime(2026-10-09T05:05:42.5848876Z);\nlet _endTime = datetime(2026-10-09T05:10:42.5848875Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T05:05:42.5848876Z);\nlet _endTime = datetime(2026-10-09T05:10:42.5848875Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:46.229]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/cosmosContent"
    }�[0m
    �[37m[23:00:46.229]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:46.437]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:46.437]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/cosmosContent"
    }�[0m
    �[37m[23:00:46.447]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:49.188]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect"
    }�[0m
    �[37m[23:00:49.244]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:49.310]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:49.336]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:49.336]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:49.336]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:49.455]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:49.591]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:49.701]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:00:49.989]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:44.8927968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\", ActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", SubActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\"}, IsPermanent:false})\nfailed to generate Cosmos content Git history: failed to process entries: failed to stage file: fork/exec /usr/bin/git: resource temporarily unavailable\n",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649"
    }�[0m
    �[37m[23:00:49.989]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:44.8927968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\", ActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", SubActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\"}, IsPermanent:false})\nfailed to generate Cosmos content Git history: failed to process entries: failed to stage file: fork/exec /usr/bin/git: resource temporarily unavailable\n"
    }�[0m
    
      query-skip-hcp-and-events: command failed: exit status 1
    output: �[37m[23:00:50.121]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126"
    }�[0m
    �[37m[23:00:53.109]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:51.6990414Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\", ActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", SubActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityStack:\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:53.110]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 70/289 failed (KQL: let _startTime = datetime(2026-10-09T04:45:50.1214504Z);\nlet _endTime = datetime(2026-10-09T04:50:50.1214503Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T04:45:50.1214504Z);\nlet _endTime = datetime(2026-10-09T04:50:50.1214503Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:53.110]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/cosmosContent"
    }�[0m
    �[37m[23:00:53.110]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:53.326]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:53.326]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/cosmosContent"
    }�[0m
    �[37m[23:00:53.335]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:59.329]�[0m �[36mINFO:�[0m �[97mCompleted�[0m �[90m{
      "commits": 294
    }�[0m
    �[37m[23:00:59.329]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect"
    }�[0m
    �[37m[23:00:59.389]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:59.462]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:59.489]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:59.489]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:59.489]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:59.614]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:59.757]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:59.878]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:01:00.172]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:51.6990414Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\", ActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", SubActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityStack:\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\"}, IsPermanent:false})",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126"
    }�[0m
    �[37m[23:01:00.173]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:51.6990414Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\", ActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", SubActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityStack:\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\"}, IsPermanent:false})"
    }�[0m
    
    {
        s: "must-gather CLI test failures:\n  query-basic: command failed: exit status 1\noutput: \x1b[37m[23:00:07.094]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452\"\n}\x1b[0m\n\x1b[37m[23:00:10.068]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:08.6161084Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\\\", ActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", SubActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \\u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:10.068]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 72/289 failed (KQL: let _startTime = datetime(2026-10-09T04:55:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T05:00:07.0938714Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T04:55:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T05:00:07.0938714Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \\\"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\\\": context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:10.102]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:10.102]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:10.310]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:10.310]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:10.320]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:20.724]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mCompleted\x1b[0m \x1b[90m{\n  \"commits\": 294\n}\x1b[0m\n\x1b[37m[23:00:20.724]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:20.786]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:20.857]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:20.886]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:20.886]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:20.886]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:21.018]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:21.198]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:21.347]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:00:21.884]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:08.6161084Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\\\", ActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", SubActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \\u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\\\"}, IsPermanent:false})\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452\"\n}\x1b[0m\n\x1b[37m[23:00:21.884]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:08.6161084Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\\\", ActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", SubActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \\u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\\\"}, IsPermanent:false})\"\n}\x1b[0m\n\n  query-skip-hcp-logs: command failed: exit status 1\noutput: \x1b[37m[23:00:22.064]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616\"\n}\x1b[0m\n\x1b[37m[23:00:24.931]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:23.5364121Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\\\", ActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", SubActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \\u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:24.931]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 54/289 failed (KQL: let _startTime = datetime(2026-10-09T03:25:22.0635262Z);\\nlet _endTime = datetime(2026-10-09T03:30:22.0635261Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:24.965]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:24.965]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:25.188]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:25.188]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:25.198]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:31.232]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mCompleted\x1b[0m \x1b[90m{\n  \"commits\": 294\n}\x1b[0m\n\x1b[37m[23:00:31.233]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:31.296]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:31.363]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:31.391]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:31.391]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:31.391]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:31.520]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:31.665]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:31.785]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:00:32.084]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:23.5364121Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\\\", ActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", SubActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \\u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\\\"}, IsPermanent:false})\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616\"\n}\x1b[0m\n\x1b[37m[23:00:32.084]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:23.5364121Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\\\", ActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", SubActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \\u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\\\"}, IsPermanent:false})\"\n}\x1b[0m\n\n  query-skip-kubernetes-events: command failed: exit status 1\noutput: \x1b[37m[23:00:32.265]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048\"\n}\x1b[0m\n\x1b[37m[23:00:35.196]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:33.7819049Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\\\", ActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", SubActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \\u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:35.196]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 73/289 failed (KQL: let _startTime = datetime(2026-10-09T05:00:32.2650857Z);\\nlet _endTime = datetime(2026-10-09T05:05:32.2650856Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:35.196]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:35.196]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:35.406]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:35.406]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:35.416]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:41.551]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mCompleted\x1b[0m \x1b[90m{\n  \"commits\": 294\n}\x1b[0m\n\x1b[37m[23:00:41.551]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:41.611]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:41.684]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:41.712]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:41.713]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:41.713]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:41.839]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:41.983]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:42.104]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:00:42.413]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:33.7819049Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\\\", ActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", SubActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \\u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\\\"}, IsPermanent:false})\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048\"\n}\x1b[0m\n\x1b[37m[23:00:42.413]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:33.7819049Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\\\", ActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", SubActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \\u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\\\"}, IsPermanent:false})\"\n}\x1b[0m\n\n  query-collect-systemd-logs: command failed: exit status 1\noutput: \x1b[37m[23:00:42.585]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649\"\n}\x1b[0m\n\x1b[37m[23:00:46.166]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:44.8927968Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\\\", ActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", SubActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \\u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:46.166]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 74/289 failed (KQL: let _startTime = datetime(2026-10-09T05:05:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T05:10:42.5848875Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T05:05:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T05:10:42.5848875Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \\\"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\\\": context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:46.229]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:46.229]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:46.437]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:46.437]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:46.447]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:49.188]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:49.244]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:49.310]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:49.336]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:49.336]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:49.336]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:49.455]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:49.591]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:49.701]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:00:49.989]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:44.8927968Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\\\", ActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", SubActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \\u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\\\"}, IsPermanent:false})\\nfailed to generate Cosmos content Git history: failed to process entries: failed to stage file: fork/exec /usr/bin/git: resource temporarily unavailable\\n\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649\"\n}\x1b[0m\n\x1b[37m[23:00:49.989]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:44.8927968Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\\\", ActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", SubActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \\u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\\\"}, IsPermanent:false})\\nfailed to generate Cosmos content Git history: failed to process entries: failed to stage file: fork/exec /usr/bin/git: resource temporarily unavailable\\n\"\n}\x1b[0m\n\n  query-skip-hcp-and-events: command failed: exit status 1\noutput: \x1b[37m[23:00:50.121]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126\"\n}\x1b[0m\n\x1b[37m[23:00:53.109]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:51.6990414Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\\\", ActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", SubActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \\u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:53.110]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 70/289 failed (KQL: let _startTime = datetime(2026-10-09T04:45:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T04:50:50.1214503Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T04:45:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T04:50:50.1214503Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \\\"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\\\": context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:53.110]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:53.110]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:53.326]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:53.326]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:53.335]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:59.329]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mCompleted\x1b[0m \x1b[90m{\n  \"commits\": 294\n}\x1b[0m\n\x1b[37m[23:00:59.329]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:59.389]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:59.462]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:59.489]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:59.489]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:59.489]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:59.614]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:59.757]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:59.878]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:01:00.172]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:51.6990414Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\\\", ActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", SubActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \\u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\\\"}, IsPermanent:false})\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126\"\n}\x1b[0m\n\x1b[37m[23:01:00.173]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:51.6990414Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\\\", ActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", SubActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \\u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\\\"}, IsPermanent:false})\"\n}\x1b[0m\n",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:101]: Timed out after 651.764s.
Expected success, but got an error:
    <*errors.errorString | 0x104322d955a0>: 
    must-gather CLI test failures:
      query-basic: command failed: exit status 1
    output: �[37m[23:00:07.094]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452"
    }�[0m
    �[37m[23:00:10.068]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:08.6161084Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\", ActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", SubActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityStack:\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:10.068]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 72/289 failed (KQL: let _startTime = datetime(2026-10-09T04:55:07.0938715Z);\nlet _endTime = datetime(2026-10-09T05:00:07.0938714Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T04:55:07.0938715Z);\nlet _endTime = datetime(2026-10-09T05:00:07.0938714Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:10.102]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-basic-3825519452/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/cosmosContent"
    }�[0m
    �[37m[23:00:10.102]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-basic-3825519452/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:10.310]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:10.310]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-basic-3825519452/cosmosContent"
    }�[0m
    �[37m[23:00:10.320]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:20.724]�[0m �[36mINFO:�[0m �[97mCompleted�[0m �[90m{
      "commits": 294
    }�[0m
    �[37m[23:00:20.724]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect"
    }�[0m
    �[37m[23:00:20.786]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:20.857]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:20.886]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:20.886]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:20.886]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:21.018]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:21.198]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:21.347]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:00:21.884]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:08.6161084Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\", ActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", SubActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityStack:\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\"}, IsPermanent:false})",
      "output": "/tmp/must-gather-cli-e2e-query-basic-3825519452"
    }�[0m
    �[37m[23:00:21.884]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:08.6161084Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\", ActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", SubActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\", ActivityStack:\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\"}, IsPermanent:false})"
    }�[0m
    
      query-skip-hcp-logs: command failed: exit status 1
    output: �[37m[23:00:22.064]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616"
    }�[0m
    �[37m[23:00:24.931]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:23.5364121Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\", ActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", SubActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:24.931]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 54/289 failed (KQL: let _startTime = datetime(2026-10-09T03:25:22.0635262Z);\nlet _endTime = datetime(2026-10-09T03:30:22.0635261Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:24.965]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/cosmosContent"
    }�[0m
    �[37m[23:00:24.965]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:25.188]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:25.188]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/cosmosContent"
    }�[0m
    �[37m[23:00:25.198]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:31.232]�[0m �[36mINFO:�[0m �[97mCompleted�[0m �[90m{
      "commits": 294
    }�[0m
    �[37m[23:00:31.233]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect"
    }�[0m
    �[37m[23:00:31.296]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:31.363]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:31.391]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:31.391]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:31.391]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:31.520]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:31.665]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:31.785]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:00:32.084]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:23.5364121Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\", ActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", SubActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\"}, IsPermanent:false})",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616"
    }�[0m
    �[37m[23:00:32.084]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:23.5364121Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\", ActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", SubActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\", ActivityStack:\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\"}, IsPermanent:false})"
    }�[0m
    
      query-skip-kubernetes-events: command failed: exit status 1
    output: �[37m[23:00:32.265]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048"
    }�[0m
    �[37m[23:00:35.196]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:33.7819049Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\", ActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", SubActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityStack:\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:35.196]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 73/289 failed (KQL: let _startTime = datetime(2026-10-09T05:00:32.2650857Z);\nlet _endTime = datetime(2026-10-09T05:05:32.2650856Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:35.196]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/cosmosContent"
    }�[0m
    �[37m[23:00:35.196]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:35.406]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:35.406]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/cosmosContent"
    }�[0m
    �[37m[23:00:35.416]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:41.551]�[0m �[36mINFO:�[0m �[97mCompleted�[0m �[90m{
      "commits": 294
    }�[0m
    �[37m[23:00:41.551]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect"
    }�[0m
    �[37m[23:00:41.611]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:41.684]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:41.712]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:41.713]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:41.713]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:41.839]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:41.983]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:42.104]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:00:42.413]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:33.7819049Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\", ActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", SubActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityStack:\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\"}, IsPermanent:false})",
      "output": "/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048"
    }�[0m
    �[37m[23:00:42.413]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:33.7819049Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\", ActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", SubActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\", ActivityStack:\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\"}, IsPermanent:false})"
    }�[0m
    
      query-collect-systemd-logs: command failed: exit status 1
    output: �[37m[23:00:42.585]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649"
    }�[0m
    �[37m[23:00:46.166]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:44.8927968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\", ActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", SubActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:46.166]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 74/289 failed (KQL: let _startTime = datetime(2026-10-09T05:05:42.5848876Z);\nlet _endTime = datetime(2026-10-09T05:10:42.5848875Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T05:05:42.5848876Z);\nlet _endTime = datetime(2026-10-09T05:10:42.5848875Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:46.229]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/cosmosContent"
    }�[0m
    �[37m[23:00:46.229]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:46.437]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:46.437]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/cosmosContent"
    }�[0m
    �[37m[23:00:46.447]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:49.188]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect"
    }�[0m
    �[37m[23:00:49.244]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:49.310]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:49.336]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:49.336]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:49.336]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:49.455]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:49.591]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:49.701]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:00:49.989]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:44.8927968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\", ActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", SubActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\"}, IsPermanent:false})\nfailed to generate Cosmos content Git history: failed to process entries: failed to stage file: fork/exec /usr/bin/git: resource temporarily unavailable\n",
      "output": "/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649"
    }�[0m
    �[37m[23:00:49.989]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:44.8927968Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\", ActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", SubActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"5d767d80-6ecf-4205-a662-cd03a907247f\", ActivityStack:\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\"}, IsPermanent:false})\nfailed to generate Cosmos content Git history: failed to process entries: failed to stage file: fork/exec /usr/bin/git: resource temporarily unavailable\n"
    }�[0m
    
      query-skip-hcp-and-events: command failed: exit status 1
    output: �[37m[23:00:50.121]�[0m �[36mINFO:�[0m �[97mgathering must-gather logs�[0m �[90m{
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126"
    }�[0m
    �[37m[23:00:53.109]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:51.6990414Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\", ActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", SubActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityStack:\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\"}, IsPermanent:false})",
      "name": "detailedInfraOrchestrationLogs"
    }�[0m
    �[37m[23:00:53.110]�[0m �[91mERROR:�[0m �[97mQuery failed�[0m �[90m{
      "err": "query \"backendControllerConditions\" time window 70/289 failed (KQL: let _startTime = datetime(2026-10-09T04:45:50.1214504Z);\nlet _endTime = datetime(2026-10-09T04:50:50.1214503Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T04:45:50.1214504Z);\nlet _endTime = datetime(2026-10-09T04:50:50.1214503Z);\ndatabase('ServiceLogs').table('backendLogs')\n| where timestamp between (_startTime .. _endTime)\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\n| where container_name == 'aro-hcp-backend'\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\n| extend controller_name = extract(\"/hcpOpenShiftControllers/([^\\\\/]+)\", 1, tostring(log.currentResourceID))\n| mv-expand condition = log.content.properties.status.conditions\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context canceled",
      "name": "backendControllerConditions"
    }�[0m
    �[37m[23:00:53.110]�[0m �[36mINFO:�[0m �[97mgenerating Cosmos content Git history�[0m �[90m{
      "input": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/custom/custom-query_cosmosResourceSnapshots.jsonl",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/cosmosContent"
    }�[0m
    �[37m[23:00:53.110]�[0m �[36mINFO:�[0m �[97mParsing log file�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/custom/custom-query_cosmosResourceSnapshots.jsonl"
    }�[0m
    �[37m[23:00:53.326]�[0m �[36mINFO:�[0m �[97mFound data dump entries�[0m �[90m{
      "count": 611,
      "files": 1
    }�[0m
    �[37m[23:00:53.326]�[0m �[36mINFO:�[0m �[97mInitializing git repository�[0m �[90m{
      "path": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/cosmosContent"
    }�[0m
    �[37m[23:00:53.335]�[0m �[36mINFO:�[0m �[97mProcessing entries and creating commits�[0m �[90m{}�[0m
    �[37m[23:00:59.329]�[0m �[36mINFO:�[0m �[97mCompleted�[0m �[90m{
      "commits": 294
    }�[0m
    �[37m[23:00:59.329]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for clusters in the resource group�[0m �[90m{
      "clusterIDs": null,
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect"
    }�[0m
    �[37m[23:00:59.389]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: cluster IDs to inspect�[0m �[90m{
      "clusterIDs": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa"
      ]
    }�[0m
    �[37m[23:00:59.462]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered infra cluster names�[0m �[90m{
      "clusterNames": [
        "ci01-j8117120-mgmt-1",
        "ci01-j8117120-svc"
      ]
    }�[0m
    �[37m[23:00:59.489]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespaces": [
        "2tbhqli0e1h6selklh84pg26u47hn0qa",
        "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
        "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
      ]
    }�[0m
    �[37m[23:00:59.489]�[0m �[36mINFO:�[0m �[97moc-adm-inspect: discovered namespaces for resource group clusters�[0m �[90m{
      "managementCluster": "ci01-j8117120-mgmt-1",
      "namespaceCount": 4
    }�[0m
    �[37m[23:00:59.489]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:59.614]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:59.757]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa"
    }�[0m
    �[37m[23:00:59.878]�[0m �[36mINFO:�[0m �[97mrunning oc-adm-inspect for namespace�[0m �[90m{
      "cluster": "ci01-j8117120-mgmt-1",
      "namespace": "ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i"
    }�[0m
    �[37m[23:01:00.172]�[0m �[91mERROR:�[0m �[97mmust-gather finished with errors; partial content may have been written�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:51.6990414Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\", ActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", SubActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityStack:\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\"}, IsPermanent:false})",
      "output": "/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126"
    }�[0m
    �[37m[23:01:00.173]�[0m �[91mERROR:�[0m �[97mcommand failed�[0m �[90m{
      "err": "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n    database('ServiceLogs').table('clustersServiceLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\n    | where isnotempty(log.cid)\n    | distinct tostring(log.cid)\n    | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n    database('ServiceLogs').table('containerLogs')\n    | where timestamp between (_startTime .. _endTime)\n    | where namespace_name has \"maestro\"\n    | extend logStr = tostring(log)\n    | where logStr has _cs_cluster_id\n    | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n    | where isnotempty(resourceId)\n    | distinct resourceId\n);\n// Infrastructure \u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n    (\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"maestro\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n        // Includes the CR reconciliation logs\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"hypershift\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    ),\n    (\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n        // Includes: work agent operations, manifest application, status feedback\n        database('ServiceLogs').table('containerLogs')\n        | where timestamp between (_startTime .. _endTime)\n        | where namespace_name has \"open-cluster-management-agent\"\n        | extend logStr = tostring(log)\n        | where logStr has _cs_cluster_id\n        | project timestamp, container_name, msg = logStr, log\n    )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-09T23:00:51.6990414Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\", ActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", SubActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"6c1e30d0-9a19-434e-a986-7816b0e42259\", ActivityStack:\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\"}, IsPermanent:false})"
    }�[0m
    
    {
        s: "must-gather CLI test failures:\n  query-basic: command failed: exit status 1\noutput: \x1b[37m[23:00:07.094]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452\"\n}\x1b[0m\n\x1b[37m[23:00:10.068]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:08.6161084Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\\\", ActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", SubActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \\u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:10.068]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 72/289 failed (KQL: let _startTime = datetime(2026-10-09T04:55:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T05:00:07.0938714Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T04:55:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T05:00:07.0938714Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \\\"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\\\": context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:10.102]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:10.102]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:10.310]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:10.310]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:10.320]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:20.724]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mCompleted\x1b[0m \x1b[90m{\n  \"commits\": 294\n}\x1b[0m\n\x1b[37m[23:00:20.724]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:20.786]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:20.857]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:20.886]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:20.886]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:20.886]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:21.018]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:21.198]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:21.347]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:00:21.884]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:08.6161084Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\\\", ActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", SubActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \\u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\\\"}, IsPermanent:false})\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-basic-3825519452\"\n}\x1b[0m\n\x1b[37m[23:00:21.884]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:07.0938715Z);\\nlet _endTime = datetime(2026-10-09T23:00:07.0938720Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:08.6161084Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0\\\", ActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", SubActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"fa9fdc45-fd1f-45a6-b92b-0c4aefd52007\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;449095d6-2662-47ac-b98c-84b1c221c4c0 ARID=fa9fdc45-fd1f-45a6-b92b-0c4aefd52007 \\u003e GW.Http.CallContext/fa9fdc45-fd1f-45a6-b92b-0c4aefd52007)\\\"}, IsPermanent:false})\"\n}\x1b[0m\n\n  query-skip-hcp-logs: command failed: exit status 1\noutput: \x1b[37m[23:00:22.064]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616\"\n}\x1b[0m\n\x1b[37m[23:00:24.931]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:23.5364121Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\\\", ActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", SubActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \\u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:24.931]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 54/289 failed (KQL: let _startTime = datetime(2026-10-09T03:25:22.0635262Z);\\nlet _endTime = datetime(2026-10-09T03:30:22.0635261Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:24.965]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:24.965]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:25.188]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:25.188]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:25.198]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:31.232]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mCompleted\x1b[0m \x1b[90m{\n  \"commits\": 294\n}\x1b[0m\n\x1b[37m[23:00:31.233]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:31.296]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:31.363]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:31.391]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:31.391]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:31.391]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:31.520]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:31.665]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:31.785]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:00:32.084]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:23.5364121Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\\\", ActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", SubActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \\u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\\\"}, IsPermanent:false})\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-logs-2729146616\"\n}\x1b[0m\n\x1b[37m[23:00:32.084]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:22.0635262Z);\\nlet _endTime = datetime(2026-10-09T23:00:22.0635266Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:23.5364121Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf\\\", ActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", SubActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;f7de9eb3-7a31-483b-a7f9-1bd521db66cf ARID=98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd \\u003e GW.Http.CallContext/98bd0dee-a5dd-4e86-b8b2-3f1c9d52f1dd)\\\"}, IsPermanent:false})\"\n}\x1b[0m\n\n  query-skip-kubernetes-events: command failed: exit status 1\noutput: \x1b[37m[23:00:32.265]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048\"\n}\x1b[0m\n\x1b[37m[23:00:35.196]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:33.7819049Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\\\", ActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", SubActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \\u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:35.196]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 73/289 failed (KQL: let _startTime = datetime(2026-10-09T05:00:32.2650857Z);\\nlet _endTime = datetime(2026-10-09T05:05:32.2650856Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:35.196]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:35.196]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:35.406]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:35.406]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:35.416]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:41.551]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mCompleted\x1b[0m \x1b[90m{\n  \"commits\": 294\n}\x1b[0m\n\x1b[37m[23:00:41.551]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:41.611]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:41.684]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:41.712]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:41.713]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:41.713]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:41.839]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:41.983]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:42.104]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:00:42.413]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:33.7819049Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\\\", ActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", SubActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \\u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\\\"}, IsPermanent:false})\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-kubernetes-events-352999048\"\n}\x1b[0m\n\x1b[37m[23:00:42.413]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:32.2650857Z);\\nlet _endTime = datetime(2026-10-09T23:00:32.2650859Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:33.7819049Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0\\\", ActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", SubActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"1cb1bb92-57cf-464f-bf00-48034d2d6a93\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;01e664e9-6d59-4a47-8248-2db95e91d8b0 ARID=1cb1bb92-57cf-464f-bf00-48034d2d6a93 \\u003e GW.Http.CallContext/1cb1bb92-57cf-464f-bf00-48034d2d6a93)\\\"}, IsPermanent:false})\"\n}\x1b[0m\n\n  query-collect-systemd-logs: command failed: exit status 1\noutput: \x1b[37m[23:00:42.585]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649\"\n}\x1b[0m\n\x1b[37m[23:00:46.166]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:44.8927968Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\\\", ActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", SubActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \\u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:46.166]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 74/289 failed (KQL: let _startTime = datetime(2026-10-09T05:05:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T05:10:42.5848875Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T05:05:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T05:10:42.5848875Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \\\"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\\\": context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:46.229]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:46.229]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:46.437]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:46.437]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:46.447]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:49.188]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:49.244]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:49.310]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:49.336]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:49.336]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:49.336]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:49.455]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:49.591]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:49.701]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:00:49.989]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:44.8927968Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\\\", ActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", SubActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \\u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\\\"}, IsPermanent:false})\\nfailed to generate Cosmos content Git history: failed to process entries: failed to stage file: fork/exec /usr/bin/git: resource temporarily unavailable\\n\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-collect-systemd-logs-3798035649\"\n}\x1b[0m\n\x1b[37m[23:00:49.989]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:42.5848876Z);\\nlet _endTime = datetime(2026-10-09T23:00:42.5848880Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:44.8927968Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb\\\", ActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", SubActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"5d767d80-6ecf-4205-a662-cd03a907247f\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;d3575990-ea88-461a-9367-5a48a83ce5bb ARID=5d767d80-6ecf-4205-a662-cd03a907247f \\u003e GW.Http.CallContext/5d767d80-6ecf-4205-a662-cd03a907247f)\\\"}, IsPermanent:false})\\nfailed to generate Cosmos content Git history: failed to process entries: failed to stage file: fork/exec /usr/bin/git: resource temporarily unavailable\\n\"\n}\x1b[0m\n\n  query-skip-hcp-and-events: command failed: exit status 1\noutput: \x1b[37m[23:00:50.121]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgathering must-gather logs\x1b[0m \x1b[90m{\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126\"\n}\x1b[0m\n\x1b[37m[23:00:53.109]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:51.6990414Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\\\", ActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", SubActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \\u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\\\"}, IsPermanent:false})\",\n  \"name\": \"detailedInfraOrchestrationLogs\"\n}\x1b[0m\n\x1b[37m[23:00:53.110]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mQuery failed\x1b[0m \x1b[90m{\n  \"err\": \"query \\\"backendControllerConditions\\\" time window 70/289 failed (KQL: let _startTime = datetime(2026-10-09T04:45:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T04:50:50.1214503Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message): failed to execute query: Op(OpQuery): Kind(KHTTPError): With query: let _startTime = datetime(2026-10-09T04:45:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T04:50:50.1214503Z);\\ndatabase('ServiceLogs').table('backendLogs')\\n| where timestamp between (_startTime .. _endTime)\\n| where cluster has_any ('ci01-j8117120-mgmt-1', 'ci01-j8117120-svc', 'ci01-j8117120-mgmt-1')\\n| where container_name == 'aro-hcp-backend'\\n| where log.msg has 'dumping resourceID' and log.msg has 'hcpOpenShiftControllers'\\n| extend controller_name = extract(\\\"/hcpOpenShiftControllers/([^\\\\\\\\/]+)\\\", 1, tostring(log.currentResourceID))\\n| mv-expand condition = log.content.properties.status.conditions\\n| project timestamp, log, cluster, condition.lastTransitionTime, controller_name, condition.type, condition.status, condition.reason, condition.message, Post \\\"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\\\": context canceled\",\n  \"name\": \"backendControllerConditions\"\n}\x1b[0m\n\x1b[37m[23:00:53.110]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mgenerating Cosmos content Git history\x1b[0m \x1b[90m{\n  \"input\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/custom/custom-query_cosmosResourceSnapshots.jsonl\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:53.110]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mParsing log file\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/custom/custom-query_cosmosResourceSnapshots.jsonl\"\n}\x1b[0m\n\x1b[37m[23:00:53.326]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mFound data dump entries\x1b[0m \x1b[90m{\n  \"count\": 611,\n  \"files\": 1\n}\x1b[0m\n\x1b[37m[23:00:53.326]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mInitializing git repository\x1b[0m \x1b[90m{\n  \"path\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/cosmosContent\"\n}\x1b[0m\n\x1b[37m[23:00:53.335]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mProcessing entries and creating commits\x1b[0m \x1b[90m{}\x1b[0m\n\x1b[37m[23:00:59.329]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mCompleted\x1b[0m \x1b[90m{\n  \"commits\": 294\n}\x1b[0m\n\x1b[37m[23:00:59.329]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for clusters in the resource group\x1b[0m \x1b[90m{\n  \"clusterIDs\": null,\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect\"\n}\x1b[0m\n\x1b[37m[23:00:59.389]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: cluster IDs to inspect\x1b[0m \x1b[90m{\n  \"clusterIDs\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:59.462]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered infra cluster names\x1b[0m \x1b[90m{\n  \"clusterNames\": [\n    \"ci01-j8117120-mgmt-1\",\n    \"ci01-j8117120-svc\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:59.489]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaces\": [\n    \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n    \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n  ]\n}\x1b[0m\n\x1b[37m[23:00:59.489]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97moc-adm-inspect: discovered namespaces for resource group clusters\x1b[0m \x1b[90m{\n  \"managementCluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespaceCount\": 4\n}\x1b[0m\n\x1b[37m[23:00:59.489]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:59.614]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/klusterlet-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:59.757]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa\"\n}\x1b[0m\n\x1b[37m[23:00:59.878]\x1b[0m \x1b[36mINFO:\x1b[0m \x1b[97mrunning oc-adm-inspect for namespace\x1b[0m \x1b[90m{\n  \"cluster\": \"ci01-j8117120-mgmt-1\",\n  \"namespace\": \"ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126/oc-adm-inspect/namespaces/ocm-arohcpci01-2tbhqli0e1h6selklh84pg26u47hn0qa-g0r0h0s1i0y5q0i\"\n}\x1b[0m\n\x1b[37m[23:01:00.172]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mmust-gather finished with errors; partial content may have been written\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:51.6990414Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\\\", ActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", SubActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \\u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\\\"}, IsPermanent:false})\",\n  \"output\": \"/tmp/must-gather-cli-e2e-query-skip-hcp-and-events-94089126\"\n}\x1b[0m\n\x1b[37m[23:01:00.173]\x1b[0m \x1b[91mERROR:\x1b[0m \x1b[97mcommand failed\x1b[0m \x1b[90m{\n  \"err\": \"failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \\\"detailedInfraOrchestrationLogs\\\" result (KQL: let _startTime = datetime(2026-10-08T23:00:50.1214504Z);\\nlet _endTime = datetime(2026-10-09T23:00:50.1214507Z);\\n// Extract cluster ID from CS logs using cid field\\nlet _cs_cluster_id = toscalar(\\n    database('ServiceLogs').table('clustersServiceLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where resource_id has 'kusto-logs-5qwwl8d6wvg6'\\n    | where isnotempty(log.cid)\\n    | distinct tostring(log.cid)\\n    | limit 1\\n);\\n// Find all Maestro resource IDs for this cluster\\nlet maestro_resource_ids = (\\n    database('ServiceLogs').table('containerLogs')\\n    | where timestamp between (_startTime .. _endTime)\\n    | where namespace_name has \\\"maestro\\\"\\n    | extend logStr = tostring(log)\\n    | where logStr has _cs_cluster_id\\n    | extend resourceId = extract(@'resourceid[=:\\\"]+([a-f0-9\\\\-]+)', 1, logStr)\\n    | where isnotempty(resourceId)\\n    | distinct resourceId\\n);\\n// Infrastructure \\u0026 Orchestration Logs - Maestro, Hypershift, and ACM Agent\\nunion\\n    (\\n        // Maestro logs: Resource bundle operations for this cluster and its associated resources\\n        // Includes: resource creation, updates, deletions, status events, gRPC communications\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"maestro\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\\n        // Includes the CR reconciliation logs\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"hypershift\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    ),\\n    (\\n        // ACM Agent logs: ManifestWork reconciliation on the managed cluster\\n        // Includes: work agent operations, manifest application, status feedback\\n        database('ServiceLogs').table('containerLogs')\\n        | where timestamp between (_startTime .. _endTime)\\n        | where namespace_name has \\\"open-cluster-management-agent\\\"\\n        | extend logStr = tostring(log)\\n        | where logStr has _cs_cluster_id\\n        | project timestamp, container_name, msg = logStr, log\\n    )\\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\\\"LimitsExceeded\\\", Message:\\\"Request is invalid and cannot be executed.\\\", Description:\\\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\\\", Type:\\\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\\\", Context:v2.ErrorContext{Timestamp:\\\"2026-10-09T23:00:51.6990414Z\\\", ServiceAlias:\\\"HCP-DEV-US-2\\\", MachineName:\\\"\\\", ProcessName:\\\"\\\", ProcessId:0, ThreadId:0, ClientRequestId:\\\"KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb\\\", ActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", SubActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityType:\\\"GW.Http.CallContext\\\", ParentActivityId:\\\"6c1e30d0-9a19-434e-a986-7816b0e42259\\\", ActivityStack:\\\"(Activity stack: CRID=KGC.execute;faa1700e-f80f-400f-b4c2-b40b4bbdddbb ARID=6c1e30d0-9a19-434e-a986-7816b0e42259 \\u003e GW.Http.CallContext/6c1e30d0-9a19-434e-a986-7816b0e42259)\\\"}, IsPermanent:false})\"\n}\x1b[0m\n",
    }

77f90 sp=0x3ae32ce77ed0 pc=0x586c752
net/http.(*Transport).startDialConnForLocked.func1()
	/usr/lib/golang/src/net/http/transport.go:1629 +0x35 fp=0x3ae32ce77fe0 sp=0x3ae32ce77f90 pc=0x586c595
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x3ae32ce77fe8 sp=0x3ae32ce77fe0 pc=0x5540381
created by net/http.(*Transport).startDialConnForLocked in goroutine 119
	/usr/lib/golang/src/net/http/transport.go:1628 +0x112

goroutine 167 gp=0x3ae32ce014a0 m=nil [semacquire]:
runtime.gopark(0x3f00400063cb02?, 0x3ae32da24402?, 0xe0?, 0xc0?, 0x3ae32d696bb8?)
	/usr/lib/golang/src/runtime/proc.go:462 +0xce fp=0x3ae32d6969c0 sp=0x3ae32d6969a0 pc=0x553818e
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:468
runtime.semacquire1(0x16f798a4, 0x0, 0x0, 0x0, 0xd)
	/usr/lib/golang/src/runtime/sema.go:192 +0x232 fp=0x3ae32d696a28 sp=0x3ae32d6969c0 pc=0x5513872
runtime.semacquire(...)
	/usr/lib/golang/src/runtime/sema.go:143
runtime.gcMarkDone()
	/usr/lib/golang/src/runtime/mgc.go:1018 +0x32 fp=0x3ae32d696ac8 sp=0x3ae32d696a28 pc=0x54d6232
runtime.gcAssistAlloc(0x3ae32ce014a0)
	/usr/lib/golang/src/runtime/mgcmark.go:647 +0x2f6 fp=0x3ae32d696b58 sp=0x3ae32d696ac8 pc=0x54dadd6
runtime.deductAssistCredit(0x1?)
	/usr/lib/golang/src/runtime/malloc.go:1861 +0x46 fp=0x3ae32d696b78 sp=0x3ae32d696b58 pc=0x54cb6a6
runtime.mallocgc(0x590, 0xf965760, 0x1)
	/usr/lib/golang/src/runtime/malloc.go:1113 +0x67 fp=0x3ae32d696ba8 sp=0x3ae32d696b78 pc=0x5534e67
runtime.newobject(0x3ae32da2445d?)
	/usr/lib/golang/src/runtime/malloc.go:2157 +0x25 fp=0x3ae32d696bd0 sp=0x3ae32d696ba8 pc=0x54cb7a5
crypto/x509.parseCertificate({0x3ae32da4c000, 0x56d, 0x582})
	/usr/lib/golang/src/crypto/x509/parser.go:887 +0x45 fp=0x3ae32d697068 sp=0x3ae32d696bd0 pc=0x5729285
crypto/x509.ParseCertificate({0x3ae32da4c000?, 0x56d, 0x2ea83?})
	/usr/lib/golang/src/crypto/x509/parser.go:1077 +0x1d fp=0x3ae32d697090 sp=0x3ae32d697068 pc=0x572a69d
crypto/x509.(*CertPool).AppendCertsFromPEM(0x3ae32d4196e0, {0x3ae32da80000?, 0x100a51a0?, 0x10059470?})
	/usr/lib/golang/src/crypto/x509/cert_pool.go:231 +0x125 fp=0x3ae32d6971b0 sp=0x3ae32d697090 pc=0x571d2c5
crypto/x509.loadSystemRoots()
	/usr/lib/golang/src/crypto/x509/root_unix.go:72 +0x391 fp=0x3ae32d697298 sp=0x3ae32d6971b0 pc=0x572ccf1
crypto/x509.initSystemRoots()
	/usr/lib/golang/src/crypto/x509/root.go:43 +0x65 fp=0x3ae32d6972d8 sp=0x3ae32d697298 pc=0x572c785
sync.(*Once).doSlow(0x54ca9ea?, 0x1f1a89e3ef160b9?)
	/usr/lib/golang/src/sync/once.go:78 +0xac fp=0x3ae32d697330 sp=0x3ae32d6972d8 pc=0x555198c
sync.(*Once).Do(...)
	/usr/lib/golang/src/sync/once.go:69
crypto/x509.systemRootsPool()
	/usr/lib/golang/src/crypto/x509/root.go:32 +0x45 fp=0x3ae32d697380 sp=0x3ae32d697330 pc=0x572c645
crypto/x509.(*Certificate).Verify(0x3ae32d4ae008, {{0x3ae32d1d9be0, 0x19}, 0x3ae32d419620, 0x0, {0xc2aa7a68520d364e, 0xb6db3cc, 0x16f76d00}, {0x0, 0x0, ...}, ...})
	/usr/lib/golang/src/crypto/x509/verify.go:578 +0x18b fp=0x3ae32d697530 sp=0x3ae32d697380 pc=0x572f5cb
crypto/tls.(*Conn).verifyServerCertificate(0x3ae32d432008, {0x3ae32d49a5a0, 0x3, 0x4})
	/usr/lib/golang/src/crypto/tls/handshake_client.go:1168 +0xb0e fp=0x3ae32d6978b8 sp=0x3ae32d697530 pc=0x5781dee
crypto/tls.(*clientHandshakeStateTLS13).readServerCertificate(0x3ae32d697c80)
	/usr/lib/golang/src/crypto/tls/handshake_client_tls13.go:640 +0x29a fp=0x3ae32d697ab0 sp=0x3ae32d6978b8 pc=0x578629a
crypto/tls.(*clientHandshakeStateTLS13).handshake(0x3ae32d697c80)
	/usr/lib/golang/src/crypto/tls/handshake_client_tls13.go:141 +0x7cc fp=0x3ae32d697b70 sp=0x3ae32d697ab0 pc=0x578344c
crypto/tls.(*Conn).clientHandshake(0x3ae32d432008, {0x1018a380, 0x3ae32ca24690})
	/usr/lib/golang/src/crypto/tls/handshake_client.go:339 +0x828 fp=0x3ae32d697d88 sp=0x3ae32d697b70 pc=0x577cb68
crypto/tls.(*Conn).clientHandshake-fm({0x1018a380?, 0x3ae32ca24690?})
	<autogenerated>:1 +0x2a fp=0x3ae32d697db0 sp=0x3ae32d697d88 pc=0x57bcaca
crypto/tls.(*Conn).handshakeContext(0x3ae32d432008, {0x1018a380, 0x3ae32c9f4460})
	/usr/lib/golang/src/crypto/tls/conn.go:1562 +0x329 fp=0x3ae32d697f70 sp=0x3ae32d697db0 pc=0x5773829
crypto/tls.(*Conn).HandshakeContext(...)
	/usr/lib/golang/src/crypto/tls/conn.go:1516
net/http.(*persistConn).addTLS.func2()
	/usr/lib/golang/src/net/http/transport.go:1736 +0x6e fp=0x3ae32d697fe0 sp=0x3ae32d697f70 pc=0x586d3ce
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x3ae32d697fe8 sp=0x3ae32d697fe0 pc=0x5540381
created by net/http.(*persistConn).addTLS in goroutine 120
	/usr/lib/golang/src/net/http/transport.go:1732 +0x309

rax    0x0
rbx    0x7fe37a988640
rcx    0x7fe3c265d94c
rdx    0x6
rdi    0xa6f0
rsi    0xa6f4
rbp    0xa6f4
rsp    0x7fe37a987790
r8     0x7fe37a987860
r9     0x7fe3c27834e0
r10    0x8
r11    0x246
r12    0x6
r13    0x7fe37a987b60
r14    0x3ae32bf97860
r15    0xffffffffffffffff
rip    0x7fe3c265d94c
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command
77f90 sp=0x3ae32ce77ed0 pc=0x586c752
net/http.(*Transport).startDialConnForLocked.func1()
	/usr/lib/golang/src/net/http/transport.go:1629 +0x35 fp=0x3ae32ce77fe0 sp=0x3ae32ce77f90 pc=0x586c595
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x3ae32ce77fe8 sp=0x3ae32ce77fe0 pc=0x5540381
created by net/http.(*Transport).startDialConnForLocked in goroutine 119
	/usr/lib/golang/src/net/http/transport.go:1628 +0x112

goroutine 167 gp=0x3ae32ce014a0 m=nil [semacquire]:
runtime.gopark(0x3f00400063cb02?, 0x3ae32da24402?, 0xe0?, 0xc0?, 0x3ae32d696bb8?)
	/usr/lib/golang/src/runtime/proc.go:462 +0xce fp=0x3ae32d6969c0 sp=0x3ae32d6969a0 pc=0x553818e
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:468
runtime.semacquire1(0x16f798a4, 0x0, 0x0, 0x0, 0xd)
	/usr/lib/golang/src/runtime/sema.go:192 +0x232 fp=0x3ae32d696a28 sp=0x3ae32d6969c0 pc=0x5513872
runtime.semacquire(...)
	/usr/lib/golang/src/runtime/sema.go:143
runtime.gcMarkDone()
	/usr/lib/golang/src/runtime/mgc.go:1018 +0x32 fp=0x3ae32d696ac8 sp=0x3ae32d696a28 pc=0x54d6232
runtime.gcAssistAlloc(0x3ae32ce014a0)
	/usr/lib/golang/src/runtime/mgcmark.go:647 +0x2f6 fp=0x3ae32d696b58 sp=0x3ae32d696ac8 pc=0x54dadd6
runtime.deductAssistCredit(0x1?)
	/usr/lib/golang/src/runtime/malloc.go:1861 +0x46 fp=0x3ae32d696b78 sp=0x3ae32d696b58 pc=0x54cb6a6
runtime.mallocgc(0x590, 0xf965760, 0x1)
	/usr/lib/golang/src/runtime/malloc.go:1113 +0x67 fp=0x3ae32d696ba8 sp=0x3ae32d696b78 pc=0x5534e67
runtime.newobject(0x3ae32da2445d?)
	/usr/lib/golang/src/runtime/malloc.go:2157 +0x25 fp=0x3ae32d696bd0 sp=0x3ae32d696ba8 pc=0x54cb7a5
crypto/x509.parseCertificate({0x3ae32da4c000, 0x56d, 0x582})
	/usr/lib/golang/src/crypto/x509/parser.go:887 +0x45 fp=0x3ae32d697068 sp=0x3ae32d696bd0 pc=0x5729285
crypto/x509.ParseCertificate({0x3ae32da4c000?, 0x56d, 0x2ea83?})
	/usr/lib/golang/src/crypto/x509/parser.go:1077 +0x1d fp=0x3ae32d697090 sp=0x3ae32d697068 pc=0x572a69d
crypto/x509.(*CertPool).AppendCertsFromPEM(0x3ae32d4196e0, {0x3ae32da80000?, 0x100a51a0?, 0x10059470?})
	/usr/lib/golang/src/crypto/x509/cert_pool.go:231 +0x125 fp=0x3ae32d6971b0 sp=0x3ae32d697090 pc=0x571d2c5
crypto/x509.loadSystemRoots()
	/usr/lib/golang/src/crypto/x509/root_unix.go:72 +0x391 fp=0x3ae32d697298 sp=0x3ae32d6971b0 pc=0x572ccf1
crypto/x509.initSystemRoots()
	/usr/lib/golang/src/crypto/x509/root.go:43 +0x65 fp=0x3ae32d6972d8 sp=0x3ae32d697298 pc=0x572c785
sync.(*Once).doSlow(0x54ca9ea?, 0x1f1a89e3ef160b9?)
	/usr/lib/golang/src/sync/once.go:78 +0xac fp=0x3ae32d697330 sp=0x3ae32d6972d8 pc=0x555198c
sync.(*Once).Do(...)
	/usr/lib/golang/src/sync/once.go:69
crypto/x509.systemRootsPool()
	/usr/lib/golang/src/crypto/x509/root.go:32 +0x45 fp=0x3ae32d697380 sp=0x3ae32d697330 pc=0x572c645
crypto/x509.(*Certificate).Verify(0x3ae32d4ae008, {{0x3ae32d1d9be0, 0x19}, 0x3ae32d419620, 0x0, {0xc2aa7a68520d364e, 0xb6db3cc, 0x16f76d00}, {0x0, 0x0, ...}, ...})
	/usr/lib/golang/src/crypto/x509/verify.go:578 +0x18b fp=0x3ae32d697530 sp=0x3ae32d697380 pc=0x572f5cb
crypto/tls.(*Conn).verifyServerCertificate(0x3ae32d432008, {0x3ae32d49a5a0, 0x3, 0x4})
	/usr/lib/golang/src/crypto/tls/handshake_client.go:1168 +0xb0e fp=0x3ae32d6978b8 sp=0x3ae32d697530 pc=0x5781dee
crypto/tls.(*clientHandshakeStateTLS13).readServerCertificate(0x3ae32d697c80)
	/usr/lib/golang/src/crypto/tls/handshake_client_tls13.go:640 +0x29a fp=0x3ae32d697ab0 sp=0x3ae32d6978b8 pc=0x578629a
crypto/tls.(*clientHandshakeStateTLS13).handshake(0x3ae32d697c80)
	/usr/lib/golang/src/crypto/tls/handshake_client_tls13.go:141 +0x7cc fp=0x3ae32d697b70 sp=0x3ae32d697ab0 pc=0x578344c
crypto/tls.(*Conn).clientHandshake(0x3ae32d432008, {0x1018a380, 0x3ae32ca24690})
	/usr/lib/golang/src/crypto/tls/handshake_client.go:339 +0x828 fp=0x3ae32d697d88 sp=0x3ae32d697b70 pc=0x577cb68
crypto/tls.(*Conn).clientHandshake-fm({0x1018a380?, 0x3ae32ca24690?})
	<autogenerated>:1 +0x2a fp=0x3ae32d697db0 sp=0x3ae32d697d88 pc=0x57bcaca
crypto/tls.(*Conn).handshakeContext(0x3ae32d432008, {0x1018a380, 0x3ae32c9f4460})
	/usr/lib/golang/src/crypto/tls/conn.go:1562 +0x329 fp=0x3ae32d697f70 sp=0x3ae32d697db0 pc=0x5773829
crypto/tls.(*Conn).HandshakeContext(...)
	/usr/lib/golang/src/crypto/tls/conn.go:1516
net/http.(*persistConn).addTLS.func2()
	/usr/lib/golang/src/net/http/transport.go:1736 +0x6e fp=0x3ae32d697fe0 sp=0x3ae32d697f70 pc=0x586d3ce
runtime.goexit({})
	/usr/lib/golang/src/runtime/asm_amd64.s:1771 +0x1 fp=0x3ae32d697fe8 sp=0x3ae32d697fe0 pc=0x5540381
created by net/http.(*persistConn).addTLS in goroutine 120
	/usr/lib/golang/src/net/http/transport.go:1732 +0x309

rax    0x0
rbx    0x7fe37a988640
rcx    0x7fe3c265d94c
rdx    0x6
rdi    0xa6f0
rsi    0xa6f4
rbp    0xa6f4
rsp    0x7fe37a987790
r8     0x7fe37a987860
r9     0x7fe3c27834e0
r10    0x8
r11    0x246
r12    0x6
r13    0x7fe37a987b60
r14    0x3ae32bf97860
r15    0xffffffffffffffff
rip    0x7fe3c265d94c
rflags 0x246
cs     0x33
fs     0x0
gs     0x0
Command Error: exit status 2
Deserialization Error: no output from command

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T23:35:06Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j8117120-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.216:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8117120/providers/microsoft.monitor/accounts/services-j8117120", namespace="klusterlet-2tbhpus9mi4ns169dgekasr6fitd43j9", pod="arohcp-monitor-kube-state-metrics-7988dc648c-bc7zx", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2tbhpus9mi4ns169dgekasr6fitd43j9 on cluster ci01-j8117120-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-rf2ccpxh56qx/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-rf2ccpxh56qx/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-j7bk7p"
Unexpected error:
    <*fmt.wrapError | 0x76fa41aa1e0>: 
    DNS for route host agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x76fa41aa1a0>{
            msg: "DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-21-j7bk7p"
Unexpected error:
    <*fmt.wrapError | 0x76fa41aa1e0>: 
    DNS for route host agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0x76fa41aa1a0>{
            msg: "DNS for agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-mjcfp.apps.aro.y0q5g2g2n5j8w7n.wl6t.j4583424.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x247e648e4440>: 
    DNS for route host agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x247e648e4420>{
            msg: "DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:158]: failed to verify simple web app runs on v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x247e648e4440>: 
    DNS for route host agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
        err: <*fmt.wrapError | 0x247e648e4420>{
            msg: "DNS for agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-9xcxj.apps.aro.v20261001.jhke.j8166656.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T21:28:57Z
  Ended:       2026-10-09T21:36:57Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8166656-svc", component="backend", name="actualhostedcluster", region="westus3", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0x3113509f94c0>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-g776ncnttgrt" to finish creating: GET https://rp.j3788032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77fe8352-5c4d-493e-9b0c-7d5a039edff5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77fe8352-5c4d-493e-9b0c-7d5a039edff5",
      "name": "77fe8352-5c4d-493e-9b0c-7d5a039edff5",
      "status": "Failed",
      "startTime": "2026-10-09T19:34:01.982763751Z",
      "endTime": "2026-10-09T20:03:10.160667249Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-5j5ks-29l5h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-29l5h to exist\nMachine ephemeral-disk-ephemeral-np-5j5ks-4jtss: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-4jtss to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-5j5ks-29l5h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-29l5h to exist\nMachine ephemeral-disk-ephemeral-np-5j5ks-4jtss: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-4jtss to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:108]: failed to create nodepool ephemeral-np with ephemeral OS disk
Unexpected error:
    <*fmt.wrapError | 0x3113509f94c0>: 
    failed to create NodePool ephemeral-np: failed waiting for nodepool="ephemeral-np" for cluster "ephemeral-disk" in resourcegroup="ephemeral-osdisk-g776ncnttgrt" to finish creating: GET https://rp.j3788032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77fe8352-5c4d-493e-9b0c-7d5a039edff5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77fe8352-5c4d-493e-9b0c-7d5a039edff5",
      "name": "77fe8352-5c4d-493e-9b0c-7d5a039edff5",
      "status": "Failed",
      "startTime": "2026-10-09T19:34:01.982763751Z",
      "endTime": "2026-10-09T20:03:10.160667249Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine ephemeral-disk-ephemeral-np-5j5ks-29l5h: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-29l5h to exist\nMachine ephemeral-disk-ephemeral-np-5j5ks-4jtss: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-4jtss to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine ephemeral-disk-ephemeral-np-5j5ks-29l5h: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-29l5h to exist\nMachine ephemeral-disk-ephemeral-np-5j5ks-4jtss: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-g776ncnttgrt--managed/providers/Microsoft.Compute/virtualMachines/ephemeral-disk-ephemeral-np-5j5ks-4jtss to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:21:13Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7317504-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.112:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7317504/providers/microsoft.monitor/accounts/hcps-j7317504", namespace="ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8xgvb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:26:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7317504-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.112:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7317504/providers/microsoft.monitor/accounts/hcps-j7317504", namespace="ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8xgvb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:27:12Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j7317504-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="csi-snapshot-controller", endpoint="http", environment="ci01", instance="10.128.64.112:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7317504/providers/microsoft.monitor/accounts/hcps-j7317504", namespace="ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster", pod="arohcp-monitor-kube-state-metrics-7988dc648c-8xgvb", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbe6sullvo3b9gl7hggkskdvnm94ofr-basic-cluster/csi-snapshot-controller has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-8jfp6hmxz2vs/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.004s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-8jfp6hmxz2vs/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:312]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test
Unexpected error:
    <*fmt.wrapError | 0x2b701c6046e0>: 
    failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-jlldrxpj8hv8" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f084bf11-c7ee-4325-bb49-cf7c84015e14
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f084bf11-c7ee-4325-bb49-cf7c84015e14",
      "name": "f084bf11-c7ee-4325-bb49-cf7c84015e14",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:43.736446438Z",
      "endTime": "2026-10-09T18:31:48.075237736Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:312]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test
Unexpected error:
    <*fmt.wrapError | 0x2b701c6046e0>: 
    failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-jlldrxpj8hv8" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f084bf11-c7ee-4325-bb49-cf7c84015e14
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f084bf11-c7ee-4325-bb49-cf7c84015e14",
      "name": "f084bf11-c7ee-4325-bb49-cf7c84015e14",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:43.736446438Z",
      "endTime": "2026-10-09T18:31:48.075237736Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0x2e4acf3c8880>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-qbq2q99hlvjq" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/84d3f991-bfdd-41a1-92f7-c169b2020c7a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/84d3f991-bfdd-41a1-92f7-c169b2020c7a",
      "name": "84d3f991-bfdd-41a1-92f7-c169b2020c7a",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:10.700624897Z",
      "endTime": "2026-10-09T18:31:17.853992528Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0x2e4acf3c8880>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-qbq2q99hlvjq" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/84d3f991-bfdd-41a1-92f7-c169b2020c7a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/84d3f991-bfdd-41a1-92f7-c169b2020c7a",
      "name": "84d3f991-bfdd-41a1-92f7-c169b2020c7a",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:10.700624897Z",
      "endTime": "2026-10-09T18:31:17.853992528Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0x2f9b37b2d6a0>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-cd2gbr7b2jq2" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6aa4636f-6812-4007-8149-5c1264bfc4d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6aa4636f-6812-4007-8149-5c1264bfc4d4",
      "name": "6aa4636f-6812-4007-8149-5c1264bfc4d4",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:14.537204498Z",
      "endTime": "2026-10-09T18:32:17.726086322Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0x2f9b37b2d6a0>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-cd2gbr7b2jq2" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6aa4636f-6812-4007-8149-5c1264bfc4d4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6aa4636f-6812-4007-8149-5c1264bfc4d4",
      "name": "6aa4636f-6812-4007-8149-5c1264bfc4d4",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:14.537204498Z",
      "endTime": "2026-10-09T18:32:17.726086322Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x3aa3a58fd700>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-r9mpb5rglfkx" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/501ef370-ff82-48d0-af1d-38c7e8e27d51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/501ef370-ff82-48d0-af1d-38c7e8e27d51",
      "name": "501ef370-ff82-48d0-af1d-38c7e8e27d51",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:09.983214688Z",
      "endTime": "2026-10-09T18:31:17.777246897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0x3aa3a58fd700>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-r9mpb5rglfkx" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/501ef370-ff82-48d0-af1d-38c7e8e27d51
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/501ef370-ff82-48d0-af1d-38c7e8e27d51",
      "name": "501ef370-ff82-48d0-af1d-38c7e8e27d51",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:09.983214688Z",
      "endTime": "2026-10-09T18:31:17.777246897Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0x1062442c6c90>: 
    GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/328cd7fa-dd20-4157-ae05-7528e1885422
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/328cd7fa-dd20-4157-ae05-7528e1885422",
      "name": "328cd7fa-dd20-4157-ae05-7528e1885422",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:24.978697652Z",
      "endTime": "2026-10-09T18:32:27.609289345Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0x1062442c6c90>: 
    GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/328cd7fa-dd20-4157-ae05-7528e1885422
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/328cd7fa-dd20-4157-ae05-7528e1885422",
      "name": "328cd7fa-dd20-4157-ae05-7528e1885422",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:24.978697652Z",
      "endTime": "2026-10-09T18:32:27.609289345Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0x2f0605478480>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-mng25fmg9dxg" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/979e4ed3-7c85-4832-a3e7-4f0677e8904e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/979e4ed3-7c85-4832-a3e7-4f0677e8904e",
      "name": "979e4ed3-7c85-4832-a3e7-4f0677e8904e",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:10.582225717Z",
      "endTime": "2026-10-09T18:31:17.588996241Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0x2f0605478480>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-mng25fmg9dxg" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/979e4ed3-7c85-4832-a3e7-4f0677e8904e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/979e4ed3-7c85-4832-a3e7-4f0677e8904e",
      "name": "979e4ed3-7c85-4832-a3e7-4f0677e8904e",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:10.582225717Z",
      "endTime": "2026-10-09T18:31:17.588996241Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0x2e42295f70c0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-hlqs8nkdm5t5" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c2dd073-54b7-4e32-bb4f-1181788dad41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c2dd073-54b7-4e32-bb4f-1181788dad41",
      "name": "9c2dd073-54b7-4e32-bb4f-1181788dad41",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:23.470629545Z",
      "endTime": "2026-10-09T18:31:27.909656234Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0x2e42295f70c0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-hlqs8nkdm5t5" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c2dd073-54b7-4e32-bb4f-1181788dad41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9c2dd073-54b7-4e32-bb4f-1181788dad41",
      "name": "9c2dd073-54b7-4e32-bb4f-1181788dad41",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:23.470629545Z",
      "endTime": "2026-10-09T18:31:27.909656234Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:161]: failed to create HCP cluster "cp-ystream-deny-pods-5-1-mz7lls" at install version 5.0
Unexpected error:
    <*fmt.wrapError | 0x3f661db12580>: 
    failed to create HCP cluster cp-ystream-deny-pods-5-1-mz7lls: failed waiting for cluster="cp-ystream-deny-pods-5-1-mz7lls" in resourcegroup="rg-cp-ystream-deny-pods-5-1-msvnld9ccv56" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2df78d12-2cc9-405d-b31e-84090ef6cee2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2df78d12-2cc9-405d-b31e-84090ef6cee2",
      "name": "2df78d12-2cc9-405d-b31e-84090ef6cee2",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:26.95261Z",
      "endTime": "2026-10-09T18:31:27.800516821Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:161]: failed to create HCP cluster "cp-ystream-deny-pods-5-1-mz7lls" at install version 5.0
Unexpected error:
    <*fmt.wrapError | 0x3f661db12580>: 
    failed to create HCP cluster cp-ystream-deny-pods-5-1-mz7lls: failed waiting for cluster="cp-ystream-deny-pods-5-1-mz7lls" in resourcegroup="rg-cp-ystream-deny-pods-5-1-msvnld9ccv56" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2df78d12-2cc9-405d-b31e-84090ef6cee2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2df78d12-2cc9-405d-b31e-84090ef6cee2",
      "name": "2df78d12-2cc9-405d-b31e-84090ef6cee2",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:26.95261Z",
      "endTime": "2026-10-09T18:31:27.800516821Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0x2c6b96d827c0>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-znrkb7wxw8xs" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b57af5b-3bbd-4b3b-bb11-9deeae9a1edb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b57af5b-3bbd-4b3b-bb11-9deeae9a1edb",
      "name": "0b57af5b-3bbd-4b3b-bb11-9deeae9a1edb",
      "status": "Failed",
      "startTime": "2026-10-09T18:41:04.746790867Z",
      "endTime": "2026-10-09T19:10:05.371290727Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0x2c6b96d827c0>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-znrkb7wxw8xs" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b57af5b-3bbd-4b3b-bb11-9deeae9a1edb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b57af5b-3bbd-4b3b-bb11-9deeae9a1edb",
      "name": "0b57af5b-3bbd-4b3b-bb11-9deeae9a1edb",
      "status": "Failed",
      "startTime": "2026-10-09T18:41:04.746790867Z",
      "endTime": "2026-10-09T19:10:05.371290727Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0x6f04a2a4280>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-2g2554z7wr65" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/18e9c5d8-01c0-415e-b048-7b0270ff6b9e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/18e9c5d8-01c0-415e-b048-7b0270ff6b9e",
      "name": "18e9c5d8-01c0-415e-b048-7b0270ff6b9e",
      "status": "Failed",
      "startTime": "2026-10-09T18:40:31.117011395Z",
      "endTime": "2026-10-09T19:09:35.538194291Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0x6f04a2a4280>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-2g2554z7wr65" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/18e9c5d8-01c0-415e-b048-7b0270ff6b9e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/18e9c5d8-01c0-415e-b048-7b0270ff6b9e",
      "name": "18e9c5d8-01c0-415e-b048-7b0270ff6b9e",
      "status": "Failed",
      "startTime": "2026-10-09T18:40:31.117011395Z",
      "endTime": "2026-10-09T19:09:35.538194291Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0x32c4d8d6b260>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-glcr4w5bpjtd" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ccad3244-2be6-4d98-b6eb-b17a3b23d168
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ccad3244-2be6-4d98-b6eb-b17a3b23d168",
      "name": "ccad3244-2be6-4d98-b6eb-b17a3b23d168",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:10.70429797Z",
      "endTime": "2026-10-09T18:31:17.793696001Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0x32c4d8d6b260>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-glcr4w5bpjtd" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ccad3244-2be6-4d98-b6eb-b17a3b23d168
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ccad3244-2be6-4d98-b6eb-b17a3b23d168",
      "name": "ccad3244-2be6-4d98-b6eb-b17a3b23d168",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:10.70429797Z",
      "endTime": "2026-10-09T18:31:17.793696001Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x86a1799a760>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-fh4m5t8744rf" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b430aa9-d99a-445d-994d-9d7059d61cf0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b430aa9-d99a-445d-994d-9d7059d61cf0",
      "name": "3b430aa9-d99a-445d-994d-9d7059d61cf0",
      "status": "Failed",
      "startTime": "2026-10-09T18:05:45.777383606Z",
      "endTime": "2026-10-09T18:34:47.613925333Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0x86a1799a760>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-fh4m5t8744rf" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b430aa9-d99a-445d-994d-9d7059d61cf0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3b430aa9-d99a-445d-994d-9d7059d61cf0",
      "name": "3b430aa9-d99a-445d-994d-9d7059d61cf0",
      "status": "Failed",
      "startTime": "2026-10-09T18:05:45.777383606Z",
      "endTime": "2026-10-09T18:34:47.613925333Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-hcccw7" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0x8c4cacb2340>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-hcccw7: failed waiting for cluster="cp-ystream-upgrade-4-21-hcccw7" in resourcegroup="rg-cp-ystream-upgrade-4-21-6xx9lxdctr9q" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/18450837-6574-44f7-89e1-1540da6d0c6d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/18450837-6574-44f7-89e1-1540da6d0c6d",
      "name": "18450837-6574-44f7-89e1-1540da6d0c6d",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:28.52780907Z",
      "endTime": "2026-10-09T18:31:37.736612105Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-hcccw7" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0x8c4cacb2340>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-hcccw7: failed waiting for cluster="cp-ystream-upgrade-4-21-hcccw7" in resourcegroup="rg-cp-ystream-upgrade-4-21-6xx9lxdctr9q" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/18450837-6574-44f7-89e1-1540da6d0c6d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/18450837-6574-44f7-89e1-1540da6d0c6d",
      "name": "18450837-6574-44f7-89e1-1540da6d0c6d",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:28.52780907Z",
      "endTime": "2026-10-09T18:31:37.736612105Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:161]: failed to create HCP cluster "cp-ystream-deny-pods-4-21-772t2t" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0x17450f274740>: 
    failed to create HCP cluster cp-ystream-deny-pods-4-21-772t2t: failed waiting for cluster="cp-ystream-deny-pods-4-21-772t2t" in resourcegroup="rg-cp-ystream-deny-pods-4-21-94fnnfh56glc" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2df55a27-9995-474e-8bc7-d399c8de9607
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2df55a27-9995-474e-8bc7-d399c8de9607",
      "name": "2df55a27-9995-474e-8bc7-d399c8de9607",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:28.540876037Z",
      "endTime": "2026-10-09T18:31:38.071799276Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:161]: failed to create HCP cluster "cp-ystream-deny-pods-4-21-772t2t" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0x17450f274740>: 
    failed to create HCP cluster cp-ystream-deny-pods-4-21-772t2t: failed waiting for cluster="cp-ystream-deny-pods-4-21-772t2t" in resourcegroup="rg-cp-ystream-deny-pods-4-21-94fnnfh56glc" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2df55a27-9995-474e-8bc7-d399c8de9607
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2df55a27-9995-474e-8bc7-d399c8de9607",
      "name": "2df55a27-9995-474e-8bc7-d399c8de9607",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:28.540876037Z",
      "endTime": "2026-10-09T18:31:38.071799276Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xa4a6ab96900>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-gqg5dldzlbg2" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/23fed6b4-c8c3-47c9-b15d-8fac0bc11c61
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/23fed6b4-c8c3-47c9-b15d-8fac0bc11c61",
      "name": "23fed6b4-c8c3-47c9-b15d-8fac0bc11c61",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:23.028824085Z",
      "endTime": "2026-10-09T18:31:27.712062038Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xa4a6ab96900>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-gqg5dldzlbg2" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/23fed6b4-c8c3-47c9-b15d-8fac0bc11c61
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/23fed6b4-c8c3-47c9-b15d-8fac0bc11c61",
      "name": "23fed6b4-c8c3-47c9-b15d-8fac0bc11c61",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:23.028824085Z",
      "endTime": "2026-10-09T18:31:27.712062038Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x63eb2f385c0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-bht6w8mbz9cj" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f21fff17-63d9-4b47-bce9-ae469eda2373
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f21fff17-63d9-4b47-bce9-ae469eda2373",
      "name": "f21fff17-63d9-4b47-bce9-ae469eda2373",
      "status": "Failed",
      "startTime": "2026-10-09T18:40:44.373195167Z",
      "endTime": "2026-10-09T19:09:45.384935766Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x63eb2f385c0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-bht6w8mbz9cj" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f21fff17-63d9-4b47-bce9-ae469eda2373
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f21fff17-63d9-4b47-bce9-ae469eda2373",
      "name": "f21fff17-63d9-4b47-bce9-ae469eda2373",
      "status": "Failed",
      "startTime": "2026-10-09T18:40:44.373195167Z",
      "endTime": "2026-10-09T19:09:45.384935766Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:92]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x199bbb710720>: 
    failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-mrtmndgzlmdf" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b44366c-80d7-439e-998d-9fcc151df343
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b44366c-80d7-439e-998d-9fcc151df343",
      "name": "7b44366c-80d7-439e-998d-9fcc151df343",
      "status": "Failed",
      "startTime": "2026-10-09T18:40:36.99482192Z",
      "endTime": "2026-10-09T19:09:45.381090093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:92]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x199bbb710720>: 
    failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-mrtmndgzlmdf" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b44366c-80d7-439e-998d-9fcc151df343
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b44366c-80d7-439e-998d-9fcc151df343",
      "name": "7b44366c-80d7-439e-998d-9fcc151df343",
      "status": "Failed",
      "startTime": "2026-10-09T18:40:36.99482192Z",
      "endTime": "2026-10-09T19:09:45.381090093Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xb3d17b32c20>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-vzwh878hqpng" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4b9f32dc-673d-478c-88d7-fc5808ea056e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4b9f32dc-673d-478c-88d7-fc5808ea056e",
      "name": "4b9f32dc-673d-478c-88d7-fc5808ea056e",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:50.079224939Z",
      "endTime": "2026-10-09T18:32:57.767672257Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xb3d17b32c20>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-vzwh878hqpng" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4b9f32dc-673d-478c-88d7-fc5808ea056e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4b9f32dc-673d-478c-88d7-fc5808ea056e",
      "name": "4b9f32dc-673d-478c-88d7-fc5808ea056e",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:50.079224939Z",
      "endTime": "2026-10-09T18:32:57.767672257Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-59rb7vv6whnw/cluster-candidate-4-23-gk696c should provision
Unexpected error:
    <*fmt.wrapError | 0x37c98825a960>: 
    failed to create HCP cluster cluster-candidate-4-23-gk696c: failed waiting for cluster="cluster-candidate-4-23-gk696c" in resourcegroup="rg-candidate-4-23-59rb7vv6whnw" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/afeb42bf-90f0-441c-a63e-aff4f3ce70ff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/afeb42bf-90f0-441c-a63e-aff4f3ce70ff",
      "name": "afeb42bf-90f0-441c-a63e-aff4f3ce70ff",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:04.181877373Z",
      "endTime": "2026-10-09T18:31:07.60472581Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-59rb7vv6whnw/cluster-candidate-4-23-gk696c should provision
Unexpected error:
    <*fmt.wrapError | 0x37c98825a960>: 
    failed to create HCP cluster cluster-candidate-4-23-gk696c: failed waiting for cluster="cluster-candidate-4-23-gk696c" in resourcegroup="rg-candidate-4-23-59rb7vv6whnw" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/afeb42bf-90f0-441c-a63e-aff4f3ce70ff
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/afeb42bf-90f0-441c-a63e-aff4f3ce70ff",
      "name": "afeb42bf-90f0-441c-a63e-aff4f3ce70ff",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:04.181877373Z",
      "endTime": "2026-10-09T18:31:07.60472581Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:432]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0x32e4652764e0>: 
    failed waiting for cluster="acr-pull" in resourcegroup="acr-pull-ghsrnfbzbs7w" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01cc1b85-cb04-4948-81af-a84ee1172f5e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01cc1b85-cb04-4948-81af-a84ee1172f5e",
      "name": "01cc1b85-cb04-4948-81af-a84ee1172f5e",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:04.712611439Z",
      "endTime": "2026-10-09T18:32:07.994233939Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:432]: failed to create HCP cluster with containerRegistry set
Unexpected error:
    <*fmt.wrapError | 0x32e4652764e0>: 
    failed waiting for cluster="acr-pull" in resourcegroup="acr-pull-ghsrnfbzbs7w" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01cc1b85-cb04-4948-81af-a84ee1172f5e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01cc1b85-cb04-4948-81af-a84ee1172f5e",
      "name": "01cc1b85-cb04-4948-81af-a84ee1172f5e",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:04.712611439Z",
      "endTime": "2026-10-09T18:32:07.994233939Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0x202a20e30bc0>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-vb2b7x9n2fn6" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d15c2d04-d306-443f-a667-d1fa4c91ef63
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d15c2d04-d306-443f-a667-d1fa4c91ef63",
      "name": "d15c2d04-d306-443f-a667-d1fa4c91ef63",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:58.922056002Z",
      "endTime": "2026-10-09T18:32:07.653066445Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics"
Unexpected error:
    <*fmt.wrapError | 0x202a20e30bc0>: 
    failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-vb2b7x9n2fn6" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d15c2d04-d306-443f-a667-d1fa4c91ef63
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d15c2d04-d306-443f-a667-d1fa4c91ef63",
      "name": "d15c2d04-d306-443f-a667-d1fa4c91ef63",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:58.922056002Z",
      "endTime": "2026-10-09T18:32:07.653066445Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:83]: failed to create HCP cluster np-update-nodes-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x3185feba73a0>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-v28bzj42qplb" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a41e5fcd-6e53-4c51-a232-f6e978e0b6f4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a41e5fcd-6e53-4c51-a232-f6e978e0b6f4",
      "name": "a41e5fcd-6e53-4c51-a232-f6e978e0b6f4",
      "status": "Failed",
      "startTime": "2026-10-09T18:41:02.593702135Z",
      "endTime": "2026-10-09T19:10:05.43294293Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:83]: failed to create HCP cluster np-update-nodes-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0x3185feba73a0>: 
    failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-v28bzj42qplb" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a41e5fcd-6e53-4c51-a232-f6e978e0b6f4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a41e5fcd-6e53-4c51-a232-f6e978e0b6f4",
      "name": "a41e5fcd-6e53-4c51-a232-f6e978e0b6f4",
      "status": "Failed",
      "startTime": "2026-10-09T18:41:02.593702135Z",
      "endTime": "2026-10-09T19:10:05.43294293Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--zjz7xzqgvzph/cluster-hs-pre-6qmm4z should provision
Unexpected error:
    <*fmt.wrapError | 0x54f1cef8360>: 
    failed to create HCP cluster cluster-hs-pre-6qmm4z: failed waiting for cluster="cluster-hs-pre-6qmm4z" in resourcegroup="rg-hs-pre--zjz7xzqgvzph" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/13b41411-132a-4a05-bba1-afb761fbeedc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/13b41411-132a-4a05-bba1-afb761fbeedc",
      "name": "13b41411-132a-4a05-bba1-afb761fbeedc",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:26.932428728Z",
      "endTime": "2026-10-09T18:31:27.824979904Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--zjz7xzqgvzph/cluster-hs-pre-6qmm4z should provision
Unexpected error:
    <*fmt.wrapError | 0x54f1cef8360>: 
    failed to create HCP cluster cluster-hs-pre-6qmm4z: failed waiting for cluster="cluster-hs-pre-6qmm4z" in resourcegroup="rg-hs-pre--zjz7xzqgvzph" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/13b41411-132a-4a05-bba1-afb761fbeedc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/13b41411-132a-4a05-bba1-afb761fbeedc",
      "name": "13b41411-132a-4a05-bba1-afb761fbeedc",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:26.932428728Z",
      "endTime": "2026-10-09T18:31:27.824979904Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0x177e38ef13a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-lsb6swwskkg2" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1669f29d-d6df-4267-9529-a0d7da468336
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1669f29d-d6df-4267-9529-a0d7da468336",
      "name": "1669f29d-d6df-4267-9529-a0d7da468336",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:51.468582358Z",
      "endTime": "2026-10-09T18:32:57.974613483Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0x177e38ef13a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-lsb6swwskkg2" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1669f29d-d6df-4267-9529-a0d7da468336
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1669f29d-d6df-4267-9529-a0d7da468336",
      "name": "1669f29d-d6df-4267-9529-a0d7da468336",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:51.468582358Z",
      "endTime": "2026-10-09T18:32:57.974613483Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:94]: failed to create HCP cluster "sre-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x1e47d8a5e100>: 
    failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-g4z4dttdrrdz" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/151550cf-073c-4dad-a405-aed2b7653edc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/151550cf-073c-4dad-a405-aed2b7653edc",
      "name": "151550cf-073c-4dad-a405-aed2b7653edc",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:24.084781048Z",
      "endTime": "2026-10-09T18:31:27.863195526Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:94]: failed to create HCP cluster "sre-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x1e47d8a5e100>: 
    failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-g4z4dttdrrdz" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/151550cf-073c-4dad-a405-aed2b7653edc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/151550cf-073c-4dad-a405-aed2b7653edc",
      "name": "151550cf-073c-4dad-a405-aed2b7653edc",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:24.084781048Z",
      "endTime": "2026-10-09T18:31:27.863195526Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:89]: failed to create HCP cluster for kusto logs test
Unexpected error:
    <*fmt.wrapError | 0x79ffe50a2a0>: 
    failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-cxp698kznkfb" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/46fe2269-499f-4834-8ab6-b664af33b5a2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/46fe2269-499f-4834-8ab6-b664af33b5a2",
      "name": "46fe2269-499f-4834-8ab6-b664af33b5a2",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:08.122092327Z",
      "endTime": "2026-10-09T18:32:17.802040827Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:89]: failed to create HCP cluster for kusto logs test
Unexpected error:
    <*fmt.wrapError | 0x79ffe50a2a0>: 
    failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-cxp698kznkfb" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/46fe2269-499f-4834-8ab6-b664af33b5a2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/46fe2269-499f-4834-8ab6-b664af33b5a2",
      "name": "46fe2269-499f-4834-8ab6-b664af33b5a2",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:08.122092327Z",
      "endTime": "2026-10-09T18:32:17.802040827Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0x35e89de92dc0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-ncqj527js7qc" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6e242d6b-6cd0-441f-9e9f-ab6a2a8cecb1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6e242d6b-6cd0-441f-9e9f-ab6a2a8cecb1",
      "name": "6e242d6b-6cd0-441f-9e9f-ab6a2a8cecb1",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:13.055503632Z",
      "endTime": "2026-10-09T18:31:18.192009212Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0x35e89de92dc0>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-ncqj527js7qc" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6e242d6b-6cd0-441f-9e9f-ab6a2a8cecb1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6e242d6b-6cd0-441f-9e9f-ab6a2a8cecb1",
      "name": "6e242d6b-6cd0-441f-9e9f-ab6a2a8cecb1",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:13.055503632Z",
      "endTime": "2026-10-09T18:31:18.192009212Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-hzq568" with version 4.23.0-ec.0 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x39d60de28c00>: 
    failed to create HCP cluster cluster-zstream-4-23-hzq568: failed waiting for cluster="cluster-zstream-4-23-hzq568" in resourcegroup="rg-zstream-upgrade-4-23-p2mrvhqrkcwk" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9025adf3-6286-4949-a9de-5c10722e6264
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9025adf3-6286-4949-a9de-5c10722e6264",
      "name": "9025adf3-6286-4949-a9de-5c10722e6264",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:19.664085411Z",
      "endTime": "2026-10-09T18:31:27.58037293Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-23-hzq568" with version 4.23.0-ec.0 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x39d60de28c00>: 
    failed to create HCP cluster cluster-zstream-4-23-hzq568: failed waiting for cluster="cluster-zstream-4-23-hzq568" in resourcegroup="rg-zstream-upgrade-4-23-p2mrvhqrkcwk" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9025adf3-6286-4949-a9de-5c10722e6264
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9025adf3-6286-4949-a9de-5c10722e6264",
      "name": "9025adf3-6286-4949-a9de-5c10722e6264",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:19.664085411Z",
      "endTime": "2026-10-09T18:31:27.58037293Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-1-b925qf4dtv4p/cluster-candidate-5-1-9pxcgq should provision
Unexpected error:
    <*fmt.wrapError | 0x2eef7967c2e0>: 
    failed to create HCP cluster cluster-candidate-5-1-9pxcgq: failed waiting for cluster="cluster-candidate-5-1-9pxcgq" in resourcegroup="rg-candidate-5-1-b925qf4dtv4p" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4fb4fa1d-4d37-4e68-9855-a3ff27e37126
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4fb4fa1d-4d37-4e68-9855-a3ff27e37126",
      "name": "4fb4fa1d-4d37-4e68-9855-a3ff27e37126",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:31.454683954Z",
      "endTime": "2026-10-09T18:31:38.135835036Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-1-b925qf4dtv4p/cluster-candidate-5-1-9pxcgq should provision
Unexpected error:
    <*fmt.wrapError | 0x2eef7967c2e0>: 
    failed to create HCP cluster cluster-candidate-5-1-9pxcgq: failed waiting for cluster="cluster-candidate-5-1-9pxcgq" in resourcegroup="rg-candidate-5-1-b925qf4dtv4p" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4fb4fa1d-4d37-4e68-9855-a3ff27e37126
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4fb4fa1d-4d37-4e68-9855-a3ff27e37126",
      "name": "4fb4fa1d-4d37-4e68-9855-a3ff27e37126",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:31.454683954Z",
      "endTime": "2026-10-09T18:31:38.135835036Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-tq5mmp4dz9tr/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-tq5mmp4dz9tr/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:421]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0x335de76525e0>: 
    failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-c656jk8nbqf7" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95424a21-7234-4d2f-b6f8-49db4fb067c9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95424a21-7234-4d2f-b6f8-49db4fb067c9",
      "name": "95424a21-7234-4d2f-b6f8-49db4fb067c9",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:19.968557469Z",
      "endTime": "2026-10-09T18:31:27.896075751Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:421]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0x335de76525e0>: 
    failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-c656jk8nbqf7" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95424a21-7234-4d2f-b6f8-49db4fb067c9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95424a21-7234-4d2f-b6f8-49db4fb067c9",
      "name": "95424a21-7234-4d2f-b6f8-49db4fb067c9",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:19.968557469Z",
      "endTime": "2026-10-09T18:31:27.896075751Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x1d7c60a347e0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-wh6shv999rvk" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d322d275-9941-45bf-a054-aa9d4b20f7cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d322d275-9941-45bf-a054-aa9d4b20f7cc",
      "name": "d322d275-9941-45bf-a054-aa9d4b20f7cc",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:10.15141187Z",
      "endTime": "2026-10-09T18:31:17.847497361Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0x1d7c60a347e0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-wh6shv999rvk" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d322d275-9941-45bf-a054-aa9d4b20f7cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d322d275-9941-45bf-a054-aa9d4b20f7cc",
      "name": "d322d275-9941-45bf-a054-aa9d4b20f7cc",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:10.15141187Z",
      "endTime": "2026-10-09T18:31:17.847497361Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x23c23ea44480>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-cxkvb78fvbzb" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6df5258-6815-41f2-b102-078ded35b588
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6df5258-6815-41f2-b102-078ded35b588",
      "name": "d6df5258-6815-41f2-b102-078ded35b588",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:31.8726861Z",
      "endTime": "2026-10-09T18:31:37.732392148Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0x23c23ea44480>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-cxkvb78fvbzb" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6df5258-6815-41f2-b102-078ded35b588
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6df5258-6815-41f2-b102-078ded35b588",
      "name": "d6df5258-6815-41f2-b102-078ded35b588",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:31.8726861Z",
      "endTime": "2026-10-09T18:31:37.732392148Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:161]: failed to create HCP cluster "cp-ystream-deny-pods-4-22-27qv8b" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x3d26d7338bc0>: 
    failed to create HCP cluster cp-ystream-deny-pods-4-22-27qv8b: failed waiting for cluster="cp-ystream-deny-pods-4-22-27qv8b" in resourcegroup="rg-cp-ystream-deny-pods-4-22-w7hdxl28d99s" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2e8a3a4-fc97-4e60-8ea1-fe9527661b48
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2e8a3a4-fc97-4e60-8ea1-fe9527661b48",
      "name": "f2e8a3a4-fc97-4e60-8ea1-fe9527661b48",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:58.253531041Z",
      "endTime": "2026-10-09T18:32:07.93537792Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:161]: failed to create HCP cluster "cp-ystream-deny-pods-4-22-27qv8b" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x3d26d7338bc0>: 
    failed to create HCP cluster cp-ystream-deny-pods-4-22-27qv8b: failed waiting for cluster="cp-ystream-deny-pods-4-22-27qv8b" in resourcegroup="rg-cp-ystream-deny-pods-4-22-w7hdxl28d99s" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2e8a3a4-fc97-4e60-8ea1-fe9527661b48
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f2e8a3a4-fc97-4e60-8ea1-fe9527661b48",
      "name": "f2e8a3a4-fc97-4e60-8ea1-fe9527661b48",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:58.253531041Z",
      "endTime": "2026-10-09T18:32:07.93537792Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-sjswpn" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x2cc77affe3a0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-sjswpn: failed waiting for cluster="cp-ystream-upgrade-5-0-sjswpn" in resourcegroup="rg-cp-ystream-upgrade-5-0-b6ql69rn2p27" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4c74b61a-0ab6-46a0-8dd1-69911db4a507
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4c74b61a-0ab6-46a0-8dd1-69911db4a507",
      "name": "4c74b61a-0ab6-46a0-8dd1-69911db4a507",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:28.035516007Z",
      "endTime": "2026-10-09T18:31:37.653027995Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-sjswpn" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x2cc77affe3a0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-sjswpn: failed waiting for cluster="cp-ystream-upgrade-5-0-sjswpn" in resourcegroup="rg-cp-ystream-upgrade-5-0-b6ql69rn2p27" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4c74b61a-0ab6-46a0-8dd1-69911db4a507
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4c74b61a-0ab6-46a0-8dd1-69911db4a507",
      "name": "4c74b61a-0ab6-46a0-8dd1-69911db4a507",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:28.035516007Z",
      "endTime": "2026-10-09T18:31:37.653027995Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc0a71da6e20>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-pckr4j464gtb" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db6745bc-bc9e-4dc4-bb47-135c48031922
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db6745bc-bc9e-4dc4-bb47-135c48031922",
      "name": "db6745bc-bc9e-4dc4-bb47-135c48031922",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:20.359791772Z",
      "endTime": "2026-10-09T18:31:27.841147423Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc0a71da6e20>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-pckr4j464gtb" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db6745bc-bc9e-4dc4-bb47-135c48031922
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db6745bc-bc9e-4dc4-bb47-135c48031922",
      "name": "db6745bc-bc9e-4dc4-bb47-135c48031922",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:20.359791772Z",
      "endTime": "2026-10-09T18:31:27.841147423Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0x67005e0a1e0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-lwp6hlccqt4d" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc683353-bd36-4308-9731-9574c60f012a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc683353-bd36-4308-9731-9574c60f012a",
      "name": "cc683353-bd36-4308-9731-9574c60f012a",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:01.339352229Z",
      "endTime": "2026-10-09T18:32:07.902911075Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0x67005e0a1e0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-lwp6hlccqt4d" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc683353-bd36-4308-9731-9574c60f012a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cc683353-bd36-4308-9731-9574c60f012a",
      "name": "cc683353-bd36-4308-9731-9574c60f012a",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:01.339352229Z",
      "endTime": "2026-10-09T18:32:07.902911075Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0x89590912720>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-bgt54n958sgj" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8af78792-3346-4e6f-9171-3d4b46deee20
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8af78792-3346-4e6f-9171-3d4b46deee20",
      "name": "8af78792-3346-4e6f-9171-3d4b46deee20",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:43.998867234Z",
      "endTime": "2026-10-09T18:31:48.180518414Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0x89590912720>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-bgt54n958sgj" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8af78792-3346-4e6f-9171-3d4b46deee20
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8af78792-3346-4e6f-9171-3d4b46deee20",
      "name": "8af78792-3346-4e6f-9171-3d4b46deee20",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:43.998867234Z",
      "endTime": "2026-10-09T18:31:48.180518414Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-g72ptm
Unexpected error:
    <*fmt.wrapError | 0xb7f60d7e220>: 
    failed to create HCP cluster np-noedge-g72ptm: failed waiting for cluster="np-noedge-g72ptm" in resourcegroup="rg-np-noedge-g72ptm-5wstl2fwg4bc" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/216a2acd-d28d-4c98-b531-56113e46cd95
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/216a2acd-d28d-4c98-b531-56113e46cd95",
      "name": "216a2acd-d28d-4c98-b531-56113e46cd95",
      "status": "Failed",
      "startTime": "2026-10-09T18:41:59.413528951Z",
      "endTime": "2026-10-09T19:11:05.371290712Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:303]: failed to create HCP cluster np-noedge-g72ptm
Unexpected error:
    <*fmt.wrapError | 0xb7f60d7e220>: 
    failed to create HCP cluster np-noedge-g72ptm: failed waiting for cluster="np-noedge-g72ptm" in resourcegroup="rg-np-noedge-g72ptm-5wstl2fwg4bc" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/216a2acd-d28d-4c98-b531-56113e46cd95
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/216a2acd-d28d-4c98-b531-56113e46cd95",
      "name": "216a2acd-d28d-4c98-b531-56113e46cd95",
      "status": "Failed",
      "startTime": "2026-10-09T18:41:59.413528951Z",
      "endTime": "2026-10-09T19:11:05.371290712Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-qwwhdd" with version 4.22.16 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x2771bf5ab60>: 
    failed to create HCP cluster cluster-zstream-4-22-qwwhdd: failed waiting for cluster="cluster-zstream-4-22-qwwhdd" in resourcegroup="rg-zstream-upgrade-4-22-kd9shtt6sxgz" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/33605275-6441-4d00-9992-c176dbc8fcf4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/33605275-6441-4d00-9992-c176dbc8fcf4",
      "name": "33605275-6441-4d00-9992-c176dbc8fcf4",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:09.774030716Z",
      "endTime": "2026-10-09T18:31:17.776541303Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-qwwhdd" with version 4.22.16 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x2771bf5ab60>: 
    failed to create HCP cluster cluster-zstream-4-22-qwwhdd: failed waiting for cluster="cluster-zstream-4-22-qwwhdd" in resourcegroup="rg-zstream-upgrade-4-22-kd9shtt6sxgz" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/33605275-6441-4d00-9992-c176dbc8fcf4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/33605275-6441-4d00-9992-c176dbc8fcf4",
      "name": "33605275-6441-4d00-9992-c176dbc8fcf4",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:09.774030716Z",
      "endTime": "2026-10-09T18:31:17.776541303Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-wb9qld with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x114d5955a980>: 
    failed to create HCP cluster np-version-upgrade-cluster-wb9qld: failed waiting for cluster="np-version-upgrade-cluster-wb9qld" in resourcegroup="rg-np-version-upgrade-wb9qld-8xxjfp5c7xrb" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a141e576-778c-4ae0-9f8d-563b92ca8c88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a141e576-778c-4ae0-9f8d-563b92ca8c88",
      "name": "a141e576-778c-4ae0-9f8d-563b92ca8c88",
      "status": "Failed",
      "startTime": "2026-10-09T18:40:53.187837968Z",
      "endTime": "2026-10-09T19:09:55.459750099Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-wb9qld with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x114d5955a980>: 
    failed to create HCP cluster np-version-upgrade-cluster-wb9qld: failed waiting for cluster="np-version-upgrade-cluster-wb9qld" in resourcegroup="rg-np-version-upgrade-wb9qld-8xxjfp5c7xrb" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a141e576-778c-4ae0-9f8d-563b92ca8c88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a141e576-778c-4ae0-9f8d-563b92ca8c88",
      "name": "a141e576-778c-4ae0-9f8d-563b92ca8c88",
      "status": "Failed",
      "startTime": "2026-10-09T18:40:53.187837968Z",
      "endTime": "2026-10-09T19:09:55.459750099Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0x1e00a251d180>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-dg68b6wdbl95" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b29c3a76-7e4d-4688-ba55-484661042381
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b29c3a76-7e4d-4688-ba55-484661042381",
      "name": "b29c3a76-7e4d-4688-ba55-484661042381",
      "status": "Failed",
      "startTime": "2026-10-09T18:43:00.581974003Z",
      "endTime": "2026-10-09T19:12:05.363558255Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0x1e00a251d180>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-dg68b6wdbl95" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b29c3a76-7e4d-4688-ba55-484661042381
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b29c3a76-7e4d-4688-ba55-484661042381",
      "name": "b29c3a76-7e4d-4688-ba55-484661042381",
      "status": "Failed",
      "startTime": "2026-10-09T18:43:00.581974003Z",
      "endTime": "2026-10-09T19:12:05.363558255Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x3c5369c50620>: 
    failed to create HCP cluster np-downgrade-rkvf2f: failed waiting for cluster="np-downgrade-rkvf2f" in resourcegroup="rg-np-downgrade-rkvf2f-662jssrrw87b" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/11078c9c-4b3d-492b-b5dd-86a31643023e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/11078c9c-4b3d-492b-b5dd-86a31643023e",
      "name": "11078c9c-4b3d-492b-b5dd-86a31643023e",
      "status": "Failed",
      "startTime": "2026-10-09T18:41:59.737756895Z",
      "endTime": "2026-10-09T19:11:05.391503232Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:582]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x3c5369c50620>: 
    failed to create HCP cluster np-downgrade-rkvf2f: failed waiting for cluster="np-downgrade-rkvf2f" in resourcegroup="rg-np-downgrade-rkvf2f-662jssrrw87b" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/11078c9c-4b3d-492b-b5dd-86a31643023e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/11078c9c-4b3d-492b-b5dd-86a31643023e",
      "name": "11078c9c-4b3d-492b-b5dd-86a31643023e",
      "status": "Failed",
      "startTime": "2026-10-09T18:41:59.737756895Z",
      "endTime": "2026-10-09T19:11:05.391503232Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-xqlqq2" with version 5.0.0-rc.4 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x31e860010700>: 
    failed to create HCP cluster cluster-zstream-5-0-xqlqq2: failed waiting for cluster="cluster-zstream-5-0-xqlqq2" in resourcegroup="rg-zstream-upgrade-5-0-24jbbvhcs69f" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/563513d2-4d5e-41f8-8927-4cdda458a533
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/563513d2-4d5e-41f8-8927-4cdda458a533",
      "name": "563513d2-4d5e-41f8-8927-4cdda458a533",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:31.119167931Z",
      "endTime": "2026-10-09T18:31:38.07679177Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-xqlqq2" with version 5.0.0-rc.4 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x31e860010700>: 
    failed to create HCP cluster cluster-zstream-5-0-xqlqq2: failed waiting for cluster="cluster-zstream-5-0-xqlqq2" in resourcegroup="rg-zstream-upgrade-5-0-24jbbvhcs69f" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/563513d2-4d5e-41f8-8927-4cdda458a533
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/563513d2-4d5e-41f8-8927-4cdda458a533",
      "name": "563513d2-4d5e-41f8-8927-4cdda458a533",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:31.119167931Z",
      "endTime": "2026-10-09T18:31:38.07679177Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xe827a362640>: 
    failed to create HCP cluster np-dg-minor-9q2848: failed waiting for cluster="np-dg-minor-9q2848" in resourcegroup="rg-np-dg-minor-9q2848-bfwtcf254248" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86d9b8b1-6079-4b0c-afab-f3603094a5e5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86d9b8b1-6079-4b0c-afab-f3603094a5e5",
      "name": "86d9b8b1-6079-4b0c-afab-f3603094a5e5",
      "status": "Failed",
      "startTime": "2026-10-09T18:41:59.2025741Z",
      "endTime": "2026-10-09T19:11:05.47797695Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:722]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xe827a362640>: 
    failed to create HCP cluster np-dg-minor-9q2848: failed waiting for cluster="np-dg-minor-9q2848" in resourcegroup="rg-np-dg-minor-9q2848-bfwtcf254248" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86d9b8b1-6079-4b0c-afab-f3603094a5e5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/86d9b8b1-6079-4b0c-afab-f3603094a5e5",
      "name": "86d9b8b1-6079-4b0c-afab-f3603094a5e5",
      "status": "Failed",
      "startTime": "2026-10-09T18:41:59.2025741Z",
      "endTime": "2026-10-09T19:11:05.47797695Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:144]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x30228660e7e0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-j7mf9xmh7lv5" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c9c95f39-63cf-434b-99f8-aff685f1dce0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c9c95f39-63cf-434b-99f8-aff685f1dce0",
      "name": "c9c95f39-63cf-434b-99f8-aff685f1dce0",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:22.972274217Z",
      "endTime": "2026-10-09T18:31:27.809908038Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:144]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0x30228660e7e0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-j7mf9xmh7lv5" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c9c95f39-63cf-434b-99f8-aff685f1dce0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c9c95f39-63cf-434b-99f8-aff685f1dce0",
      "name": "c9c95f39-63cf-434b-99f8-aff685f1dce0",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:22.972274217Z",
      "endTime": "2026-10-09T18:31:27.809908038Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x13eb0e2191a0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-bkn7fgqj44mp" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-bkn7fgqj44mp\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x13eb0e00ddf0>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0x13eb0e2191a0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-bkn7fgqj44mp" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-bkn7fgqj44mp\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x13eb0e00ddf0>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-bkn7fgqj44mp",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-tq7rmml6kns7/cluster-candidate-4-21-m4tzmh should provision
Unexpected error:
    <*fmt.wrapError | 0x16bbe4e5c040>: 
    failed to create HCP cluster cluster-candidate-4-21-m4tzmh: failed waiting for cluster="cluster-candidate-4-21-m4tzmh" in resourcegroup="rg-candidate-4-21-tq7rmml6kns7" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6529b35-2e31-464a-9bd0-fb3d59b6eb58
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6529b35-2e31-464a-9bd0-fb3d59b6eb58",
      "name": "d6529b35-2e31-464a-9bd0-fb3d59b6eb58",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:19.926181985Z",
      "endTime": "2026-10-09T18:31:27.888156177Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-tq7rmml6kns7/cluster-candidate-4-21-m4tzmh should provision
Unexpected error:
    <*fmt.wrapError | 0x16bbe4e5c040>: 
    failed to create HCP cluster cluster-candidate-4-21-m4tzmh: failed waiting for cluster="cluster-candidate-4-21-m4tzmh" in resourcegroup="rg-candidate-4-21-tq7rmml6kns7" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6529b35-2e31-464a-9bd0-fb3d59b6eb58
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d6529b35-2e31-464a-9bd0-fb3d59b6eb58",
      "name": "d6529b35-2e31-464a-9bd0-fb3d59b6eb58",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:19.926181985Z",
      "endTime": "2026-10-09T18:31:27.888156177Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0x1cc812df71e0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-dfnkz2xl6z58" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/695cdd08-958c-4e2f-983d-9e12b25bf415
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/695cdd08-958c-4e2f-983d-9e12b25bf415",
      "name": "695cdd08-958c-4e2f-983d-9e12b25bf415",
      "status": "Failed",
      "startTime": "2026-10-09T18:38:35.022304297Z",
      "endTime": "2026-10-09T19:07:35.444195011Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0x1cc812df71e0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-dfnkz2xl6z58" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/695cdd08-958c-4e2f-983d-9e12b25bf415
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/695cdd08-958c-4e2f-983d-9e12b25bf415",
      "name": "695cdd08-958c-4e2f-983d-9e12b25bf415",
      "status": "Failed",
      "startTime": "2026-10-09T18:38:35.022304297Z",
      "endTime": "2026-10-09T19:07:35.444195011Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:161]: failed to create HCP cluster "cp-ystream-deny-pods-5-0-92mvl6" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x2def1bfb6400>: 
    failed to create HCP cluster cp-ystream-deny-pods-5-0-92mvl6: failed waiting for cluster="cp-ystream-deny-pods-5-0-92mvl6" in resourcegroup="rg-cp-ystream-deny-pods-5-0-mbjsvxff6788" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a366c997-6f86-4a02-b79f-33520fc978fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a366c997-6f86-4a02-b79f-33520fc978fd",
      "name": "a366c997-6f86-4a02-b79f-33520fc978fd",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:55.520160493Z",
      "endTime": "2026-10-09T18:31:58.019865769Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:161]: failed to create HCP cluster "cp-ystream-deny-pods-5-0-92mvl6" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x2def1bfb6400>: 
    failed to create HCP cluster cp-ystream-deny-pods-5-0-92mvl6: failed waiting for cluster="cp-ystream-deny-pods-5-0-92mvl6" in resourcegroup="rg-cp-ystream-deny-pods-5-0-mbjsvxff6788" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a366c997-6f86-4a02-b79f-33520fc978fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a366c997-6f86-4a02-b79f-33520fc978fd",
      "name": "a366c997-6f86-4a02-b79f-33520fc978fd",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:55.520160493Z",
      "endTime": "2026-10-09T18:31:58.019865769Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-gv78x8dzj7jg/cluster-candidate-4-22-cf6vkf should provision
Unexpected error:
    <*fmt.wrapError | 0x8447b69e100>: 
    failed to create HCP cluster cluster-candidate-4-22-cf6vkf: failed waiting for cluster="cluster-candidate-4-22-cf6vkf" in resourcegroup="rg-candidate-4-22-gv78x8dzj7jg" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d732952-987f-47ff-98c9-4b6fc716c2d0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d732952-987f-47ff-98c9-4b6fc716c2d0",
      "name": "8d732952-987f-47ff-98c9-4b6fc716c2d0",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:30.381493598Z",
      "endTime": "2026-10-09T18:31:38.035050636Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-gv78x8dzj7jg/cluster-candidate-4-22-cf6vkf should provision
Unexpected error:
    <*fmt.wrapError | 0x8447b69e100>: 
    failed to create HCP cluster cluster-candidate-4-22-cf6vkf: failed waiting for cluster="cluster-candidate-4-22-cf6vkf" in resourcegroup="rg-candidate-4-22-gv78x8dzj7jg" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d732952-987f-47ff-98c9-4b6fc716c2d0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d732952-987f-47ff-98c9-4b6fc716c2d0",
      "name": "8d732952-987f-47ff-98c9-4b6fc716c2d0",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:30.381493598Z",
      "endTime": "2026-10-09T18:31:38.035050636Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:98]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xee8e0b28e60>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-2lrhvqr2hpk6" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24f458ec-25ac-4ad4-8d4f-bb9cf5274285
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24f458ec-25ac-4ad4-8d4f-bb9cf5274285",
      "name": "24f458ec-25ac-4ad4-8d4f-bb9cf5274285",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:09.548074629Z",
      "endTime": "2026-10-09T18:32:17.843512898Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:98]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xee8e0b28e60>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-2lrhvqr2hpk6" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24f458ec-25ac-4ad4-8d4f-bb9cf5274285
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/24f458ec-25ac-4ad4-8d4f-bb9cf5274285",
      "name": "24f458ec-25ac-4ad4-8d4f-bb9cf5274285",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:09.548074629Z",
      "endTime": "2026-10-09T18:32:17.843512898Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0x3969761f4860>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-2wqwxnq2ff6g" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3905ee8d-705b-4fc0-9d67-f7496f5b640c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3905ee8d-705b-4fc0-9d67-f7496f5b640c",
      "name": "3905ee8d-705b-4fc0-9d67-f7496f5b640c",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:29.879376585Z",
      "endTime": "2026-10-09T18:31:37.637229023Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0x3969761f4860>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-2wqwxnq2ff6g" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3905ee8d-705b-4fc0-9d67-f7496f5b640c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3905ee8d-705b-4fc0-9d67-f7496f5b640c",
      "name": "3905ee8d-705b-4fc0-9d67-f7496f5b640c",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:29.879376585Z",
      "endTime": "2026-10-09T18:31:37.637229023Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0x6e954516aa0>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-ch4snv2mkn8l" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/963657bb-3c12-49eb-aad6-6ef799365ba8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/963657bb-3c12-49eb-aad6-6ef799365ba8",
      "name": "963657bb-3c12-49eb-aad6-6ef799365ba8",
      "status": "Failed",
      "startTime": "2026-10-09T18:42:48.756939438Z",
      "endTime": "2026-10-09T19:11:55.377380525Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0x6e954516aa0>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-ch4snv2mkn8l" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/963657bb-3c12-49eb-aad6-6ef799365ba8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/963657bb-3c12-49eb-aad6-6ef799365ba8",
      "name": "963657bb-3c12-49eb-aad6-6ef799365ba8",
      "status": "Failed",
      "startTime": "2026-10-09T18:42:48.756939438Z",
      "endTime": "2026-10-09T19:11:55.377380525Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapErrors | 0x7f2f57b95f0>: 
    failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-hwcbq22lzw9l" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-hwcbq22lzw9l\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x7f2f606c4c0>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l",
            },
            <*fmt.wrapErrors | 0x7f2f57b95c0>{
                msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-hwcbq22lzw9l\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0x7f2f606c4c0>{
                        s: "timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapErrors | 0x7f2f57b95f0>: 
    failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-hwcbq22lzw9l" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cluster-np-labels-taints, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-hwcbq22lzw9l\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0x7f2f606c4c0>{
                s: "timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l",
            },
            <*fmt.wrapErrors | 0x7f2f57b95c0>{
                msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-hwcbq22lzw9l\" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0x7f2f606c4c0>{
                        s: "timeout '30.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-np-labels-taints in resource group rg-np-labels-taints-hwcbq22lzw9l",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-shl6mv" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc608374a6e0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-shl6mv: failed waiting for cluster="cp-ystream-upgrade-4-23-shl6mv" in resourcegroup="rg-cp-ystream-upgrade-4-23-4cxlmv6b22n7" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4792cc94-7e2d-4032-bd4f-acb61864439e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4792cc94-7e2d-4032-bd4f-acb61864439e",
      "name": "4792cc94-7e2d-4032-bd4f-acb61864439e",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:29.794897918Z",
      "endTime": "2026-10-09T18:31:38.092037296Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-shl6mv" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc608374a6e0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-shl6mv: failed waiting for cluster="cp-ystream-upgrade-4-23-shl6mv" in resourcegroup="rg-cp-ystream-upgrade-4-23-4cxlmv6b22n7" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4792cc94-7e2d-4032-bd4f-acb61864439e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4792cc94-7e2d-4032-bd4f-acb61864439e",
      "name": "4792cc94-7e2d-4032-bd4f-acb61864439e",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:29.794897918Z",
      "endTime": "2026-10-09T18:31:38.092037296Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0x3bdd1b9e300>: 
    failed to create HCP cluster ea-listjxcfch: failed waiting for cluster="ea-listjxcfch" in resourcegroup="ea-list-bwj9cp8sqnd5" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c8059bfb-fd3f-4e24-b492-e6ec0c766279
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c8059bfb-fd3f-4e24-b492-e6ec0c766279",
      "name": "c8059bfb-fd3f-4e24-b492-e6ec0c766279",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:06.045432669Z",
      "endTime": "2026-10-09T18:32:07.780692137Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0x3bdd1b9e300>: 
    failed to create HCP cluster ea-listjxcfch: failed waiting for cluster="ea-listjxcfch" in resourcegroup="ea-list-bwj9cp8sqnd5" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c8059bfb-fd3f-4e24-b492-e6ec0c766279
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c8059bfb-fd3f-4e24-b492-e6ec0c766279",
      "name": "c8059bfb-fd3f-4e24-b492-e6ec0c766279",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:06.045432669Z",
      "endTime": "2026-10-09T18:32:07.780692137Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:161]: failed to create HCP cluster "cp-ystream-deny-pods-4-23-kvt24g" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x273a184a5d60>: 
    failed to create HCP cluster cp-ystream-deny-pods-4-23-kvt24g: failed waiting for cluster="cp-ystream-deny-pods-4-23-kvt24g" in resourcegroup="rg-cp-ystream-deny-pods-4-23-ftdlqcp8hmwk" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b70812c-ce11-40d5-b792-3d06ddb946ed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b70812c-ce11-40d5-b792-3d06ddb946ed",
      "name": "6b70812c-ce11-40d5-b792-3d06ddb946ed",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:06.839137502Z",
      "endTime": "2026-10-09T18:32:07.991193612Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:161]: failed to create HCP cluster "cp-ystream-deny-pods-4-23-kvt24g" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0x273a184a5d60>: 
    failed to create HCP cluster cp-ystream-deny-pods-4-23-kvt24g: failed waiting for cluster="cp-ystream-deny-pods-4-23-kvt24g" in resourcegroup="rg-cp-ystream-deny-pods-4-23-ftdlqcp8hmwk" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b70812c-ce11-40d5-b792-3d06ddb946ed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6b70812c-ce11-40d5-b792-3d06ddb946ed",
      "name": "6b70812c-ce11-40d5-b792-3d06ddb946ed",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:06.839137502Z",
      "endTime": "2026-10-09T18:32:07.991193612Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0x3f8cfca18160>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-d9vjk88cdxmv" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/715e9b71-ab21-4169-9e2e-f3f804cb80c0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/715e9b71-ab21-4169-9e2e-f3f804cb80c0",
      "name": "715e9b71-ab21-4169-9e2e-f3f804cb80c0",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:40.105550226Z",
      "endTime": "2026-10-09T18:31:47.946839601Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0x3f8cfca18160>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-d9vjk88cdxmv" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/715e9b71-ab21-4169-9e2e-f3f804cb80c0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/715e9b71-ab21-4169-9e2e-f3f804cb80c0",
      "name": "715e9b71-ab21-4169-9e2e-f3f804cb80c0",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:40.105550226Z",
      "endTime": "2026-10-09T18:31:47.946839601Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-hkw56n" with version 4.20.40 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x149ef4a0e680>: 
    failed to create HCP cluster cluster-zstream-4-20-hkw56n: failed waiting for cluster="cluster-zstream-4-20-hkw56n" in resourcegroup="rg-zstream-upgrade-4-20-sp6t2zkkxgkt" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e9add752-dc4d-4949-a0b4-888b3fea1164
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e9add752-dc4d-4949-a0b4-888b3fea1164",
      "name": "e9add752-dc4d-4949-a0b4-888b3fea1164",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:27.811481675Z",
      "endTime": "2026-10-09T18:31:27.878574533Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-hkw56n" with version 4.20.40 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x149ef4a0e680>: 
    failed to create HCP cluster cluster-zstream-4-20-hkw56n: failed waiting for cluster="cluster-zstream-4-20-hkw56n" in resourcegroup="rg-zstream-upgrade-4-20-sp6t2zkkxgkt" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e9add752-dc4d-4949-a0b4-888b3fea1164
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e9add752-dc4d-4949-a0b4-888b3fea1164",
      "name": "e9add752-dc4d-4949-a0b4-888b3fea1164",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:27.811481675Z",
      "endTime": "2026-10-09T18:31:27.878574533Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x301e62396440>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-z7jw7nqhsrrj" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dda75f8f-4faa-422a-ac16-0ae5a3cb1048
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dda75f8f-4faa-422a-ac16-0ae5a3cb1048",
      "name": "dda75f8f-4faa-422a-ac16-0ae5a3cb1048",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:44.098063409Z",
      "endTime": "2026-10-09T18:31:48.324258901Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0x301e62396440>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-z7jw7nqhsrrj" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dda75f8f-4faa-422a-ac16-0ae5a3cb1048
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dda75f8f-4faa-422a-ac16-0ae5a3cb1048",
      "name": "dda75f8f-4faa-422a-ac16-0ae5a3cb1048",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:44.098063409Z",
      "endTime": "2026-10-09T18:31:48.324258901Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-r8nmpd with version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x2571f3e86580>: 
    failed to create HCP cluster np-version-upgrade-cluster-r8nmpd: failed waiting for cluster="np-version-upgrade-cluster-r8nmpd" in resourcegroup="rg-np-version-upgrade-r8nmpd-gs94bsxjkcq8" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3582a7c9-6838-4b54-983d-f11c1e0c0069
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3582a7c9-6838-4b54-983d-f11c1e0c0069",
      "name": "3582a7c9-6838-4b54-983d-f11c1e0c0069",
      "status": "Failed",
      "startTime": "2026-10-09T18:41:58.482094136Z",
      "endTime": "2026-10-09T19:11:05.361143305Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-r8nmpd with version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x2571f3e86580>: 
    failed to create HCP cluster np-version-upgrade-cluster-r8nmpd: failed waiting for cluster="np-version-upgrade-cluster-r8nmpd" in resourcegroup="rg-np-version-upgrade-r8nmpd-gs94bsxjkcq8" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3582a7c9-6838-4b54-983d-f11c1e0c0069
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3582a7c9-6838-4b54-983d-f11c1e0c0069",
      "name": "3582a7c9-6838-4b54-983d-f11c1e0c0069",
      "status": "Failed",
      "startTime": "2026-10-09T18:41:58.482094136Z",
      "endTime": "2026-10-09T19:11:05.361143305Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-q98qg9" at install version 5.0
Unexpected error:
    <*fmt.wrapError | 0x6049f4b8880>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-q98qg9: failed waiting for cluster="cp-ystream-upgrade-5-1-q98qg9" in resourcegroup="rg-cp-ystream-upgrade-5-1-wnb9rxjktgkl" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1c38232c-6d4d-40dc-b43c-a5f7246c3ad8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1c38232c-6d4d-40dc-b43c-a5f7246c3ad8",
      "name": "1c38232c-6d4d-40dc-b43c-a5f7246c3ad8",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:29.626443008Z",
      "endTime": "2026-10-09T18:31:38.091422062Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-q98qg9" at install version 5.0
Unexpected error:
    <*fmt.wrapError | 0x6049f4b8880>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-q98qg9: failed waiting for cluster="cp-ystream-upgrade-5-1-q98qg9" in resourcegroup="rg-cp-ystream-upgrade-5-1-wnb9rxjktgkl" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1c38232c-6d4d-40dc-b43c-a5f7246c3ad8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1c38232c-6d4d-40dc-b43c-a5f7246c3ad8",
      "name": "1c38232c-6d4d-40dc-b43c-a5f7246c3ad8",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:29.626443008Z",
      "endTime": "2026-10-09T18:31:38.091422062Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-4ppqcm with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x1e64447d2540>: 
    failed to create HCP cluster np-version-upgrade-cluster-4ppqcm: failed waiting for cluster="np-version-upgrade-cluster-4ppqcm" in resourcegroup="rg-np-version-upgrade-4ppqcm-d4stdt7h7cd7" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2e73d1f6-3086-4b9b-9df2-64fd1cd60773
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2e73d1f6-3086-4b9b-9df2-64fd1cd60773",
      "name": "2e73d1f6-3086-4b9b-9df2-64fd1cd60773",
      "status": "Failed",
      "startTime": "2026-10-09T18:40:53.127528563Z",
      "endTime": "2026-10-09T19:09:55.435737413Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-4ppqcm with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x1e64447d2540>: 
    failed to create HCP cluster np-version-upgrade-cluster-4ppqcm: failed waiting for cluster="np-version-upgrade-cluster-4ppqcm" in resourcegroup="rg-np-version-upgrade-4ppqcm-d4stdt7h7cd7" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2e73d1f6-3086-4b9b-9df2-64fd1cd60773
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2e73d1f6-3086-4b9b-9df2-64fd1cd60773",
      "name": "2e73d1f6-3086-4b9b-9df2-64fd1cd60773",
      "status": "Failed",
      "startTime": "2026-10-09T18:40:53.127528563Z",
      "endTime": "2026-10-09T19:09:55.435737413Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-4q2hm4p8flft/cluster-candidate-5-0-7p85mb should provision
Unexpected error:
    <*fmt.wrapError | 0x29bfd3f10a60>: 
    failed to create HCP cluster cluster-candidate-5-0-7p85mb: failed waiting for cluster="cluster-candidate-5-0-7p85mb" in resourcegroup="rg-candidate-5-0-4q2hm4p8flft" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf5d31f8-b2b3-4e7f-95ac-accad89f8a22
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf5d31f8-b2b3-4e7f-95ac-accad89f8a22",
      "name": "cf5d31f8-b2b3-4e7f-95ac-accad89f8a22",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:10.48921674Z",
      "endTime": "2026-10-09T18:31:17.829305884Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-4q2hm4p8flft/cluster-candidate-5-0-7p85mb should provision
Unexpected error:
    <*fmt.wrapError | 0x29bfd3f10a60>: 
    failed to create HCP cluster cluster-candidate-5-0-7p85mb: failed waiting for cluster="cluster-candidate-5-0-7p85mb" in resourcegroup="rg-candidate-5-0-4q2hm4p8flft" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf5d31f8-b2b3-4e7f-95ac-accad89f8a22
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cf5d31f8-b2b3-4e7f-95ac-accad89f8a22",
      "name": "cf5d31f8-b2b3-4e7f-95ac-accad89f8a22",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:10.48921674Z",
      "endTime": "2026-10-09T18:31:17.829305884Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-mz6c7q2wdqbz/cluster-candidate-4-20-5nddfd should provision
Unexpected error:
    <*fmt.wrapError | 0x17af8efbc1e0>: 
    failed to create HCP cluster cluster-candidate-4-20-5nddfd: failed waiting for cluster="cluster-candidate-4-20-5nddfd" in resourcegroup="rg-candidate-4-20-mz6c7q2wdqbz" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e696d5b3-02d4-4f4a-8473-f34742a9e396
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e696d5b3-02d4-4f4a-8473-f34742a9e396",
      "name": "e696d5b3-02d4-4f4a-8473-f34742a9e396",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:00.235189165Z",
      "endTime": "2026-10-09T18:32:07.930455779Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-mz6c7q2wdqbz/cluster-candidate-4-20-5nddfd should provision
Unexpected error:
    <*fmt.wrapError | 0x17af8efbc1e0>: 
    failed to create HCP cluster cluster-candidate-4-20-5nddfd: failed waiting for cluster="cluster-candidate-4-20-5nddfd" in resourcegroup="rg-candidate-4-20-mz6c7q2wdqbz" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e696d5b3-02d4-4f4a-8473-f34742a9e396
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e696d5b3-02d4-4f4a-8473-f34742a9e396",
      "name": "e696d5b3-02d4-4f4a-8473-f34742a9e396",
      "status": "Failed",
      "startTime": "2026-10-09T18:03:00.235189165Z",
      "endTime": "2026-10-09T18:32:07.930455779Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-v4mqdl at version 4.22.17
Unexpected error:
    <*fmt.wrapError | 0x39d668b22540>: 
    failed to create HCP cluster np-skip-minor-v4mqdl: failed waiting for cluster="np-skip-minor-v4mqdl" in resourcegroup="rg-np-skip-minor-v4mqdl-mjk8skltxp9z" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/88d993e2-c096-42e9-a517-e34a00ecb286
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/88d993e2-c096-42e9-a517-e34a00ecb286",
      "name": "88d993e2-c096-42e9-a517-e34a00ecb286",
      "status": "Failed",
      "startTime": "2026-10-09T18:41:59.152989858Z",
      "endTime": "2026-10-09T19:11:05.472167684Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:443]: failed to create HCP cluster np-skip-minor-v4mqdl at version 4.22.17
Unexpected error:
    <*fmt.wrapError | 0x39d668b22540>: 
    failed to create HCP cluster np-skip-minor-v4mqdl: failed waiting for cluster="np-skip-minor-v4mqdl" in resourcegroup="rg-np-skip-minor-v4mqdl-mjk8skltxp9z" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/88d993e2-c096-42e9-a517-e34a00ecb286
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/88d993e2-c096-42e9-a517-e34a00ecb286",
      "name": "88d993e2-c096-42e9-a517-e34a00ecb286",
      "status": "Failed",
      "startTime": "2026-10-09T18:41:59.152989858Z",
      "endTime": "2026-10-09T19:11:05.472167684Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-4sfvrk" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x3edb50de6460>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-4sfvrk: failed waiting for cluster="cp-ystream-upgrade-4-22-4sfvrk" in resourcegroup="rg-cp-ystream-upgrade-4-22-2msl55xkftls" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e56901b2-67f6-4317-a7ca-ba4d115c9a3b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e56901b2-67f6-4317-a7ca-ba4d115c9a3b",
      "name": "e56901b2-67f6-4317-a7ca-ba4d115c9a3b",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:17.464807841Z",
      "endTime": "2026-10-09T18:31:17.868854868Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-4sfvrk" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0x3edb50de6460>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-4sfvrk: failed waiting for cluster="cp-ystream-upgrade-4-22-4sfvrk" in resourcegroup="rg-cp-ystream-upgrade-4-22-2msl55xkftls" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e56901b2-67f6-4317-a7ca-ba4d115c9a3b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e56901b2-67f6-4317-a7ca-ba4d115c9a3b",
      "name": "e56901b2-67f6-4317-a7ca-ba4d115c9a3b",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:17.464807841Z",
      "endTime": "2026-10-09T18:31:17.868854868Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-79n8sz" with version 4.21.35 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x2331d37ea500>: 
    failed to create HCP cluster cluster-zstream-4-21-79n8sz: failed waiting for cluster="cluster-zstream-4-21-79n8sz" in resourcegroup="rg-zstream-upgrade-4-21-dc4tt6ftvkcp" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70dc33e8-b20d-4994-a031-48efd910a40d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70dc33e8-b20d-4994-a031-48efd910a40d",
      "name": "70dc33e8-b20d-4994-a031-48efd910a40d",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:09.770843031Z",
      "endTime": "2026-10-09T18:31:17.79586721Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-21-79n8sz" with version 4.21.35 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0x2331d37ea500>: 
    failed to create HCP cluster cluster-zstream-4-21-79n8sz: failed waiting for cluster="cluster-zstream-4-21-79n8sz" in resourcegroup="rg-zstream-upgrade-4-21-dc4tt6ftvkcp" to finish creating: GET https://rp.j1936896.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70dc33e8-b20d-4994-a031-48efd910a40d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/70dc33e8-b20d-4994-a031-48efd910a40d",
      "name": "70dc33e8-b20d-4994-a031-48efd910a40d",
      "status": "Failed",
      "startTime": "2026-10-09T18:02:09.770843031Z",
      "endTime": "2026-10-09T18:31:17.79586721Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [defaultIngressServingCertificate] default ingress certificate not yet synced"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:17:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1936896-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1936896/providers/microsoft.monitor/accounts/hcps-j1936896", namespace="ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frw78", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:17:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1936896-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1936896/providers/microsoft.monitor/accounts/hcps-j1936896", namespace="ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frw78", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:17:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1936896-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1936896/providers/microsoft.monitor/accounts/hcps-j1936896", namespace="ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frw78", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:17:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1936896-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.193:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1936896/providers/microsoft.monitor/accounts/hcps-j1936896", namespace="ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q", pod="arohcp-monitor-kube-state-metrics-7988dc648c-frw78", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tbdsg0oi92tdeeesp9b3hjkkb21uj1g-i5m2d9v6b0l3v4q/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T18:38:22Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterProvisionErrors1h5m", cluster="ci01-j1936896-svc", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1936896/providers/microsoft.monitor/accounts/services-j1936896", region="westus3", severity="3", short_window="5m", slo="cluster-provision-errors"
  Description: More than 72% of completed cluster create (install) operations on ci01-j1936896-svc failed over the last hour with at least 3 failures, a 14.4x burn of the 95% SLO budget.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T19:03:21Z
  Severity:    Sev3
  Labels:      alertname="userJourneyClusterProvisionErrors6h30m", cluster="ci01-j1936896-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1936896/providers/microsoft.monitor/accounts/services-j1936896", region="westus3", severity="3", short_window="30m", slo="cluster-provision-errors"
  Description: More than 30% of completed cluster create (install) operations on ci01-j1936896-svc failed over the last 6 hours (at least 5 completions), a 6x burn of the 95% SLO budget.

time=2026-10-09T16:49:10.013Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 description="Step mitigations\n Kind: Helm\n"
time=2026-10-09T16:49:10.529Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 values="map[azureCnsCpuLimits:map[enabled:true] jobImage:mcr.microsoft.com/aks/command/runtime@sha256:37df117f1179d78fd372fd35878d53ff30dfb350fb571fb197f3a2bee48bd157]"
time=2026-10-09T16:49:10.529Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1
time=2026-10-09T16:49:11.005Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 namespace=aro-hcp-mitigations
time=2026-10-09T16:49:11.316Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 dryRun=false
time=2026-10-09T16:49:11.316Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1
time=2026-10-09T16:49:11.444Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1
time=2026-10-09T16:55:10.359Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 err="failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-09T16:55:10.360Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-2xlsbr with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x2653bb4a2240>: 
    failed to create HCP cluster np-version-upgrade-cluster-2xlsbr: failed waiting for cluster="np-version-upgrade-cluster-2xlsbr" in resourcegroup="rg-np-version-upgrade-2xlsbr-hc74hngnvw7h" to finish creating: GET https://rp.j5021952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c",
      "name": "9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c",
      "status": "Failed",
      "startTime": "2026-10-09T17:24:44.640587758Z",
      "endTime": "2026-10-09T17:53:46.503674235Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-2xlsbr with version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x2653bb4a2240>: 
    failed to create HCP cluster np-version-upgrade-cluster-2xlsbr: failed waiting for cluster="np-version-upgrade-cluster-2xlsbr" in resourcegroup="rg-np-version-upgrade-2xlsbr-hc74hngnvw7h" to finish creating: GET https://rp.j5021952.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c",
      "name": "9aa7f9a2-8731-49d2-9f2d-e0d6dc9de68c",
      "status": "Failed",
      "startTime": "2026-10-09T17:24:44.640587758Z",
      "endTime": "2026-10-09T17:53:46.503674235Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T15:56:36Z
  Ended:       2026-10-09T16:18:35Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5582720-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

time=2026-10-09T15:19:24.628Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 description="Step mitigations\n Kind: Helm\n"
time=2026-10-09T15:19:25.146Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 values="map[azureCnsCpuLimits:map[enabled:true] jobImage:mcr.microsoft.com/aks/command/runtime@sha256:37df117f1179d78fd372fd35878d53ff30dfb350fb571fb197f3a2bee48bd157]"
time=2026-10-09T15:19:25.146Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-09T15:19:25.687Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 namespace=aro-hcp-mitigations
time=2026-10-09T15:19:25.942Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 dryRun=false
time=2026-10-09T15:19:25.942Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-09T15:19:26.093Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2
time=2026-10-09T15:25:12.869Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 err="failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"
time=2026-10-09T15:25:12.869Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=mitigations stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed post-install: resource Job/aro-hcp-mitigations/azure-cns-cpu-limits-patch not ready. status: Failed, message: Job Failed. failed: 1/1"

alert fired 94 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-x96nb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-x96nb has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-jtlkd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-jtlkd has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-9zkgt", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-9zkgt has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-sp5cx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-sp5cx has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-cls45", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-cls45 has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-v5hjd", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-v5hjd has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-s8kmb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-s8kmb has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-7wbzr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-7wbzr has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-g6r2m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-g6r2m has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-rq6n8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-rq6n8 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-m29g5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-m29g5 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-pcttg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-pcttg has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-kfj5n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-kfj5n has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-8mwsk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-8mwsk has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-rdzwz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-rdzwz has been in a non-ready state for longer than 5 minutes.
Firing 16:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-9nrsr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-9nrsr has been in a non-ready state for longer than 5 minutes.
Firing 17:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-bbf4f", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-bbf4f has been in a non-ready state for longer than 5 minutes.
Firing 18:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-xz7fq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-xz7fq has been in a non-ready state for longer than 5 minutes.
Firing 19:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-khnm2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-khnm2 has been in a non-ready state for longer than 5 minutes.
Firing 20:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-tn9tb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-tn9tb has been in a non-ready state for longer than 5 minutes.
Firing 21:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-cks5z", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-cks5z has been in a non-ready state for longer than 5 minutes.
Firing 22:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-lgjxz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-lgjxz has been in a non-ready state for longer than 5 minutes.
Firing 23:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-ghjjp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-ghjjp has been in a non-ready state for longer than 5 minutes.
Firing 24:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-lfpsq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-lfpsq has been in a non-ready state for longer than 5 minutes.
Firing 25:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-j9pqn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-j9pqn has been in a non-ready state for longer than 5 minutes.
Firing 26:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:05:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-5tnr6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-5tnr6 has been in a non-ready state for longer than 5 minutes.
Firing 27:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-zhh5s", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-zhh5s has been in a non-ready state for longer than 5 minutes.
Firing 28:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-df2m9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-df2m9 has been in a non-ready state for longer than 5 minutes.
Firing 29:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-d8cbr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-d8cbr has been in a non-ready state for longer than 5 minutes.
Firing 30:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-q4bf8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-q4bf8 has been in a non-ready state for longer than 5 minutes.
Firing 31:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-8vkk6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-8vkk6 has been in a non-ready state for longer than 5 minutes.
Firing 32:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-ltf9m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-ltf9m has been in a non-ready state for longer than 5 minutes.
Firing 33:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-pqcmf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-pqcmf has been in a non-ready state for longer than 5 minutes.
Firing 34:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-qddb4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-qddb4 has been in a non-ready state for longer than 5 minutes.
Firing 35:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-rxqr4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-rxqr4 has been in a non-ready state for longer than 5 minutes.
Firing 36:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-8x4dn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-8x4dn has been in a non-ready state for longer than 5 minutes.
Firing 37:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-r4p99", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-r4p99 has been in a non-ready state for longer than 5 minutes.
Firing 38:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-62kzb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-62kzb has been in a non-ready state for longer than 5 minutes.
Firing 39:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-rmkv4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-rmkv4 has been in a non-ready state for longer than 5 minutes.
Firing 40:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-r2blk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-r2blk has been in a non-ready state for longer than 5 minutes.
Firing 41:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:05:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-mzd6d", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-mzd6d has been in a non-ready state for longer than 5 minutes.
Firing 42:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-6qk79", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-6qk79 has been in a non-ready state for longer than 5 minutes.
Firing 43:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-ldl4b", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-ldl4b has been in a non-ready state for longer than 5 minutes.
Firing 44:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-skcks", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-skcks has been in a non-ready state for longer than 5 minutes.
Firing 45:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-2wnf8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-2wnf8 has been in a non-ready state for longer than 5 minutes.
Firing 46:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-g9jdz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-g9jdz has been in a non-ready state for longer than 5 minutes.
Firing 47:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-zqv6n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-zqv6n has been in a non-ready state for longer than 5 minutes.
Firing 48:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-57999", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-57999 has been in a non-ready state for longer than 5 minutes.
Firing 49:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-qvtrf", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-qvtrf has been in a non-ready state for longer than 5 minutes.
Firing 50:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-dvwv9", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-dvwv9 has been in a non-ready state for longer than 5 minutes.
Firing 51:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-dncfj", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-dncfj has been in a non-ready state for longer than 5 minutes.
Firing 52:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:05:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-9q52l", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-9q52l has been in a non-ready state for longer than 5 minutes.
Firing 53:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-5qtxq", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-5qtxq has been in a non-ready state for longer than 5 minutes.
Firing 54:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:05:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-sfjfx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-sfjfx has been in a non-ready state for longer than 5 minutes.
Firing 55:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-4vqv7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-4vqv7 has been in a non-ready state for longer than 5 minutes.
Firing 56:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-pvmct", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-pvmct has been in a non-ready state for longer than 5 minutes.
Firing 57:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-sgbt2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-sgbt2 has been in a non-ready state for longer than 5 minutes.
Firing 58:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-mtw9g", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-mtw9g has been in a non-ready state for longer than 5 minutes.
Firing 59:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-nzfck", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-nzfck has been in a non-ready state for longer than 5 minutes.
Firing 60:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:38Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-6n2zh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-6n2zh has been in a non-ready state for longer than 5 minutes.
Firing 61:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-gl292", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-gl292 has been in a non-ready state for longer than 5 minutes.
Firing 62:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-ptg2s", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-ptg2s has been in a non-ready state for longer than 5 minutes.
Firing 63:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-qvtv4", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-qvtv4 has been in a non-ready state for longer than 5 minutes.
Firing 64:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-njvdk", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-njvdk has been in a non-ready state for longer than 5 minutes.
Firing 65:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-5cqkh", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-5cqkh has been in a non-ready state for longer than 5 minutes.
Firing 66:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-t8tsp", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-t8tsp has been in a non-ready state for longer than 5 minutes.
Firing 67:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-9mtq6", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-9mtq6 has been in a non-ready state for longer than 5 minutes.
Firing 68:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:05:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-qr84n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-qr84n has been in a non-ready state for longer than 5 minutes.
Firing 69:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-nxw6l", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-nxw6l has been in a non-ready state for longer than 5 minutes.
Firing 70:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-rstls", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-rstls has been in a non-ready state for longer than 5 minutes.
Firing 71:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-vgqmv", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-vgqmv has been in a non-ready state for longer than 5 minutes.
Firing 72:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-2c8gz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-2c8gz has been in a non-ready state for longer than 5 minutes.
Firing 73:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-9q7pr", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-9q7pr has been in a non-ready state for longer than 5 minutes.
Firing 74:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-g7zj8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-g7zj8 has been in a non-ready state for longer than 5 minutes.
Firing 75:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-c9rv5", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-c9rv5 has been in a non-ready state for longer than 5 minutes.
Firing 76:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-p6wbm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-p6wbm has been in a non-ready state for longer than 5 minutes.
Firing 77:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-hrfnw", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-hrfnw has been in a non-ready state for longer than 5 minutes.
Firing 78:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:05:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-47xk7", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-47xk7 has been in a non-ready state for longer than 5 minutes.
Firing 79:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-mspcg", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-mspcg has been in a non-ready state for longer than 5 minutes.
Firing 80:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-qn65l", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-qn65l has been in a non-ready state for longer than 5 minutes.
Firing 81:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-v4hhz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-v4hhz has been in a non-ready state for longer than 5 minutes.
Firing 82:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-q4dtm", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-q4dtm has been in a non-ready state for longer than 5 minutes.
Firing 83:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-cs2fn", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-cs2fn has been in a non-ready state for longer than 5 minutes.
Firing 84:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-tv9qb", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-tv9qb has been in a non-ready state for longer than 5 minutes.
Firing 85:
  State:       Resolved
  Started:     2026-10-09T15:42:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-mtp6n", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-mtp6n has been in a non-ready state for longer than 5 minutes.
Firing 86:
  State:       Resolved
  Started:     2026-10-09T15:42:37Z
  Ended:       2026-10-09T16:09:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-s249v", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-s249v has been in a non-ready state for longer than 5 minutes.
Firing 87:
  State:       Resolved
  Started:     2026-10-09T15:43:35Z
  Ended:       2026-10-09T16:05:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-wmvms", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-wmvms has been in a non-ready state for longer than 5 minutes.
Firing 88:
  State:       Resolved
  Started:     2026-10-09T15:44:36Z
  Ended:       2026-10-09T16:02:37Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-vrj6w", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-vrj6w has been in a non-ready state for longer than 5 minutes.
Firing 89:
  State:       Resolved
  Started:     2026-10-09T15:44:36Z
  Ended:       2026-10-09T16:05:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-zggw8", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-zggw8 has been in a non-ready state for longer than 5 minutes.
Firing 90:
  State:       Resolved
  Started:     2026-10-09T15:44:36Z
  Ended:       2026-10-09T16:05:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-5rf8m", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-5rf8m has been in a non-ready state for longer than 5 minutes.
Firing 91:
  State:       Resolved
  Started:     2026-10-09T15:44:37Z
  Ended:       2026-10-09T16:05:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-8p7wx", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-8p7wx has been in a non-ready state for longer than 5 minutes.
Firing 92:
  State:       Resolved
  Started:     2026-10-09T15:44:37Z
  Ended:       2026-10-09T16:05:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="network-node-identity-8449556c45-fvbcc", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/network-node-identity-8449556c45-fvbcc has been in a non-ready state for longer than 5 minutes.
Firing 93:
  State:       Resolved
  Started:     2026-10-09T15:44:37Z
  Ended:       2026-10-09T16:02:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull", pod="csi-snapshot-controller-8464dbc457-jmhw2", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbf0ua1b1efhug6lvler0bmqlsj7ee-acr-pull/csi-snapshot-controller-8464dbc457-jmhw2 has been in a non-ready state for longer than 5 minutes.
Firing 94:
  State:       Resolved
  Started:     2026-10-09T15:44:37Z
  Ended:       2026-10-09T16:05:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6361216-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b", pod="kube-controller-manager-7d8f4c9476-7t9rz", region="westus3", severity="warning"
  Description: Pod ocm-arohcpci01-2tbbemu3kdg6s9hj0na4fbg472co0emt-n5e3l6d0m8j9y7b/kube-controller-manager-7d8f4c9476-7t9rz has been in a non-ready state for longer than 5 minutes.

time=2026-10-09T15:06:37.853Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 description="Step deploy\n Kind: Helm\n"
time=2026-10-09T15:06:38.448Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 values="map[captureMode:all clusterName:ci01-j4897024-mgmt-2 enabled:true environment:ci01 image:map[digest:sha256:11b789c31d680e7f988ca6ecb8901408801a582f560ea82572d969f5ddb243a6 registry:arohcpsvcdev.azurecr.io repository:ci-op-crcb6znc/pipeline] region:westus3 resources:map[limits:map[memory:256Mi] requests:map[cpu:20m memory:64Mi]] serviceAccount:map[name:swift-recorder]]"
time=2026-10-09T15:06:38.449Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2
time=2026-10-09T15:06:38.710Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 namespace=swift-recorder
time=2026-10-09T15:06:39.018Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 dryRun=false
time=2026-10-09T15:06:39.018Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2
time=2026-10-09T15:06:39.140Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2
time=2026-10-09T15:16:42.762Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 err="resource DaemonSet/swift-recorder/swift-recorder not ready. status: InProgress, message: Available: 16/17\ncontext deadline exceeded"
time=2026-10-09T15:16:42.762Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SwiftRecorder resourceGroup=management step=deploy stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/swift-recorder/swift-recorder not ready. status: InProgress, message: Available: 16/17\ncontext deadline exceeded"

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:20:44Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.64.99:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4897024/providers/microsoft.monitor/accounts/services-j4897024", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mfswx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:20:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.64.99:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4897024/providers/microsoft.monitor/accounts/services-j4897024", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mfswx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:20:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="ama-metrics-node", endpoint="http", environment="ci01", instance="10.128.64.99:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4897024/providers/microsoft.monitor/accounts/services-j4897024", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mfswx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/ama-metrics-node has not finished or progressed for at least 15 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:20:45Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="aks-secrets-store-csi-driver", endpoint="http", environment="ci01", instance="10.128.64.99:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4897024/providers/microsoft.monitor/accounts/services-j4897024", namespace="kube-system", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mfswx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet kube-system/aks-secrets-store-csi-driver has not finished or progressed for at least 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T15:02:04Z
  Ended:       2026-10-09T15:12:09Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j4897024-mgmt-1", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:11:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-23980774-vmss000000-slpqz", region="westus3", severity="warning"
  Description: Pod kube-system/eraser-aks-system-23980774-vmss000000-slpqz has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:11:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-lzv6r", region="westus3", severity="warning"
  Description: Pod kube-system/aks-secrets-store-csi-driver-lzv6r has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:11:41Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-qbqjf", region="westus3", severity="warning"
  Description: Pod kube-system/ama-metrics-node-qbqjf has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-09T15:12:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4897024-mgmt-2", component="kubernetes-infrastructure", namespace="swift-recorder", pod="swift-recorder-6f6gx", region="westus3", severity="warning"
  Description: Pod swift-recorder/swift-recorder-6f6gx has been in a non-ready state for longer than 5 minutes.

time=2026-10-09T14:05:11.912Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-10-09T14:35:11.381Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-10-09T14:05:11.922891129Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5137536-mgmt-2\",\"clusterName\":\"ci01-j5137536-mgmt-2\",\"region\":\"westus3\"}\n{\"time\":\"2026-10-09T14:05:14.313036396Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5137536-mgmt-2\",\"clusterName\":\"ci01-j5137536-mgmt-2\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-10-09T14:05:14.653688404Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5137536-mgmt-2\",\"clusterName\":\"ci01-j5137536-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T14:26:58Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j5137536-mgmt-1", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0xfc204541c00>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-j844k7kwk994" to finish creating: GET https://rp.j1306368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47be8315-7c7b-4770-bede-fbd3e338f83d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47be8315-7c7b-4770-bede-fbd3e338f83d",
      "name": "47be8315-7c7b-4770-bede-fbd3e338f83d",
      "status": "Failed",
      "startTime": "2026-10-09T14:07:16.315050699Z",
      "endTime": "2026-10-09T14:36:17.115068925Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c8a6f8l6z1i8v1z-cordon-np-szltg-ptx4m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-j844k7kwk994--managed/providers/Microsoft.Compute/virtualMachines/c8a6f8l6z1i8v1z-cordon-np-szltg-ptx4m to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c8a6f8l6z1i8v1z-cordon-np-szltg-ptx4m: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-j844k7kwk994--managed/providers/Microsoft.Compute/virtualMachines/c8a6f8l6z1i8v1z-cordon-np-szltg-ptx4m to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0xfc204541c00>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-j844k7kwk994" to finish creating: GET https://rp.j1306368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47be8315-7c7b-4770-bede-fbd3e338f83d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/47be8315-7c7b-4770-bede-fbd3e338f83d",
      "name": "47be8315-7c7b-4770-bede-fbd3e338f83d",
      "status": "Failed",
      "startTime": "2026-10-09T14:07:16.315050699Z",
      "endTime": "2026-10-09T14:36:17.115068925Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine c8a6f8l6z1i8v1z-cordon-np-szltg-ptx4m: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-j844k7kwk994--managed/providers/Microsoft.Compute/virtualMachines/c8a6f8l6z1i8v1z-cordon-np-szltg-ptx4m to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine c8a6f8l6z1i8v1z-cordon-np-szltg-ptx4m: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-j844k7kwk994--managed/providers/Microsoft.Compute/virtualMachines/c8a6f8l6z1i8v1z-cordon-np-szltg-ptx4m to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x87a272e80c0>: 
    resource group "autoscaling-cluster-dch49t4bvhmd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-dch49t4bvhmd" to finish deleting: GET https://rp.j2706688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/314e6611-87cc-467f-b3e8-5c7bded86eeb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/314e6611-87cc-467f-b3e8-5c7bded86eeb",
      "name": "314e6611-87cc-467f-b3e8-5c7bded86eeb",
      "status": "Failed",
      "startTime": "2026-10-09T12:11:02.340187908Z",
      "endTime": "2026-10-09T12:35:03.752595501Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x87a272e80c0>: 
    resource group "autoscaling-cluster-dch49t4bvhmd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-dch49t4bvhmd" to finish deleting: GET https://rp.j2706688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/314e6611-87cc-467f-b3e8-5c7bded86eeb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/314e6611-87cc-467f-b3e8-5c7bded86eeb",
      "name": "314e6611-87cc-467f-b3e8-5c7bded86eeb",
      "status": "Failed",
      "startTime": "2026-10-09T12:11:02.340187908Z",
      "endTime": "2026-10-09T12:35:03.752595501Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2403780381b0>: 
    resource group "rg-candidate-4-22-2twmlnh2pntd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-97w8j7" in resourcegroup="rg-candidate-4-22-2twmlnh2pntd" to finish deleting: GET https://rp.j2706688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec134b00-506f-4598-8f43-9039b6949cbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec134b00-506f-4598-8f43-9039b6949cbf",
      "name": "ec134b00-506f-4598-8f43-9039b6949cbf",
      "status": "Failed",
      "startTime": "2026-10-09T12:19:56.634862294Z",
      "endTime": "2026-10-09T12:44:04.410412622Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2403780381b0>: 
    resource group "rg-candidate-4-22-2twmlnh2pntd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-97w8j7" in resourcegroup="rg-candidate-4-22-2twmlnh2pntd" to finish deleting: GET https://rp.j2706688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec134b00-506f-4598-8f43-9039b6949cbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec134b00-506f-4598-8f43-9039b6949cbf",
      "name": "ec134b00-506f-4598-8f43-9039b6949cbf",
      "status": "Failed",
      "startTime": "2026-10-09T12:19:56.634862294Z",
      "endTime": "2026-10-09T12:44:04.410412622Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3f620761c090>: 
    resource group "customer-rg-lbqh9vzm8skr": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-lbqh9vzm8skr" to finish deleting: GET https://rp.j2706688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff9dce59-52f3-4407-ad07-f8872d204253
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff9dce59-52f3-4407-ad07-f8872d204253",
      "name": "ff9dce59-52f3-4407-ad07-f8872d204253",
      "status": "Failed",
      "startTime": "2026-10-09T12:06:09.323431375Z",
      "endTime": "2026-10-09T12:30:13.733939626Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3f620761c090>: 
    resource group "customer-rg-lbqh9vzm8skr": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-lbqh9vzm8skr" to finish deleting: GET https://rp.j2706688.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff9dce59-52f3-4407-ad07-f8872d204253
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff9dce59-52f3-4407-ad07-f8872d204253",
      "name": "ff9dce59-52f3-4407-ad07-f8872d204253",
      "status": "Failed",
      "startTime": "2026-10-09T12:06:09.323431375Z",
      "endTime": "2026-10-09T12:30:13.733939626Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T12:59:03Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2706688-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.243:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2706688/providers/microsoft.monitor/accounts/hcps-j2706688", namespace="ocm-arohcpci01-2tb8dp1rrd7cqhltc59rk3l5npdng5m4-f9l2t6z5w8u0h6h", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xtmdj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tb8dp1rrd7cqhltc59rk3l5npdng5m4-f9l2t6z5w8u0h6h/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T13:00:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2706688-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.243:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2706688/providers/microsoft.monitor/accounts/hcps-j2706688", namespace="ocm-arohcpci01-2tb8dp1rrd7cqhltc59rk3l5npdng5m4-f9l2t6z5w8u0h6h", pod="arohcp-monitor-kube-state-metrics-7988dc648c-xtmdj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tb8dp1rrd7cqhltc59rk3l5npdng5m4-f9l2t6z5w8u0h6h/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x14de9654a318>: 
    resource group "arm64-vm-cluster-bbmplmj7xnjh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-bbmplmj7xnjh" to finish deleting: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8789fff8-eb02-48e7-adfa-7a5bedc2a9ed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8789fff8-eb02-48e7-adfa-7a5bedc2a9ed",
      "name": "8789fff8-eb02-48e7-adfa-7a5bedc2a9ed",
      "status": "Failed",
      "startTime": "2026-10-09T12:10:18.071494213Z",
      "endTime": "2026-10-09T12:34:22.504221789Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x14de9654a318>: 
    resource group "arm64-vm-cluster-bbmplmj7xnjh": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-bbmplmj7xnjh" to finish deleting: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8789fff8-eb02-48e7-adfa-7a5bedc2a9ed
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8789fff8-eb02-48e7-adfa-7a5bedc2a9ed",
      "name": "8789fff8-eb02-48e7-adfa-7a5bedc2a9ed",
      "status": "Failed",
      "startTime": "2026-10-09T12:10:18.071494213Z",
      "endTime": "2026-10-09T12:34:22.504221789Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x58c914fe040>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-4zl2jn5xrcmj" to finish creating: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4fdf8129-ec4e-4f54-a67c-3179e8542d77
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4fdf8129-ec4e-4f54-a67c-3179e8542d77",
      "name": "4fdf8129-ec4e-4f54-a67c-3179e8542d77",
      "status": "Failed",
      "startTime": "2026-10-09T12:01:50.770272194Z",
      "endTime": "2026-10-09T12:30:51.906223646Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-qf6rt-prsfz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-4zl2jn5xrcmj--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-qf6rt-prsfz to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-qf6rt-prsfz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-4zl2jn5xrcmj--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-qf6rt-prsfz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:105]: failed to create node pool "np-1" for v20261001preview cluster "v20261001"
Unexpected error:
    <*fmt.wrapError | 0x58c914fe040>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "v20261001" in resourcegroup="v20261001preview-4zl2jn5xrcmj" to finish creating: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4fdf8129-ec4e-4f54-a67c-3179e8542d77
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4fdf8129-ec4e-4f54-a67c-3179e8542d77",
      "name": "4fdf8129-ec4e-4f54-a67c-3179e8542d77",
      "status": "Failed",
      "startTime": "2026-10-09T12:01:50.770272194Z",
      "endTime": "2026-10-09T12:30:51.906223646Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine v20261001-np-1-qf6rt-prsfz: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-4zl2jn5xrcmj--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-qf6rt-prsfz to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine v20261001-np-1-qf6rt-prsfz: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20261001preview-4zl2jn5xrcmj--managed/providers/Microsoft.Compute/virtualMachines/v20261001-np-1-qf6rt-prsfz to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x2ae2e5f963c0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-2c747kttkrf7": PUT https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2c747kttkrf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x2ae2e5f963c0>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-2c747kttkrf7": PUT https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-2c747kttkrf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xdad5eeb8138>: 
    resource group "external-auth-cluster-cmj7tlkjf2f4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-cmj7tlkjf2f4" to finish deleting: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a7900d0f-9832-43df-824c-157a3931804e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a7900d0f-9832-43df-824c-157a3931804e",
      "name": "a7900d0f-9832-43df-824c-157a3931804e",
      "status": "Failed",
      "startTime": "2026-10-09T12:11:18.997805773Z",
      "endTime": "2026-10-09T12:35:21.856574986Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xdad5eeb8138>: 
    resource group "external-auth-cluster-cmj7tlkjf2f4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-cluster" in resourcegroup="external-auth-cluster-cmj7tlkjf2f4" to finish deleting: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a7900d0f-9832-43df-824c-157a3931804e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a7900d0f-9832-43df-824c-157a3931804e",
      "name": "a7900d0f-9832-43df-824c-157a3931804e",
      "status": "Failed",
      "startTime": "2026-10-09T12:11:18.997805773Z",
      "endTime": "2026-10-09T12:35:21.856574986Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x354d3c777620>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-78sxml" in resourcegroup="rg-np-skip-minor-78sxml-jbc8sq5wwff6" to finish creating: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4cdfa78a-a3ff-4b30-a50e-b80eb52d2843
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4cdfa78a-a3ff-4b30-a50e-b80eb52d2843",
      "name": "4cdfa78a-a3ff-4b30-a50e-b80eb52d2843",
      "status": "Failed",
      "startTime": "2026-10-09T12:30:48.28195188Z",
      "endTime": "2026-10-09T12:59:50.51236808Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d0c9p8p4u4g1h6f-nps-420-422-9z6pl-fdr4n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-78sxml-jbc8sq5wwff6-np-sm-78sxml--managed/providers/Microsoft.Compute/virtualMachines/d0c9p8p4u4g1h6f-nps-420-422-9z6pl-fdr4n to exist\nMachine d0c9p8p4u4g1h6f-nps-420-422-9z6pl-kmv4b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-78sxml-jbc8sq5wwff6-np-sm-78sxml--managed/providers/Microsoft.Compute/virtualMachines/d0c9p8p4u4g1h6f-nps-420-422-9z6pl-kmv4b to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d0c9p8p4u4g1h6f-nps-420-422-9z6pl-fdr4n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-78sxml-jbc8sq5wwff6-np-sm-78sxml--managed/providers/Microsoft.Compute/virtualMachines/d0c9p8p4u4g1h6f-nps-420-422-9z6pl-fdr4n to exist\nMachine d0c9p8p4u4g1h6f-nps-420-422-9z6pl-kmv4b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-78sxml-jbc8sq5wwff6-np-sm-78sxml--managed/providers/Microsoft.Compute/virtualMachines/d0c9p8p4u4g1h6f-nps-420-422-9z6pl-kmv4b to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:461]: failed to create nodepool nps-420-422 at version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x354d3c777620>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-78sxml" in resourcegroup="rg-np-skip-minor-78sxml-jbc8sq5wwff6" to finish creating: GET https://rp.j0054912.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4cdfa78a-a3ff-4b30-a50e-b80eb52d2843
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4cdfa78a-a3ff-4b30-a50e-b80eb52d2843",
      "name": "4cdfa78a-a3ff-4b30-a50e-b80eb52d2843",
      "status": "Failed",
      "startTime": "2026-10-09T12:30:48.28195188Z",
      "endTime": "2026-10-09T12:59:50.51236808Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine d0c9p8p4u4g1h6f-nps-420-422-9z6pl-fdr4n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-78sxml-jbc8sq5wwff6-np-sm-78sxml--managed/providers/Microsoft.Compute/virtualMachines/d0c9p8p4u4g1h6f-nps-420-422-9z6pl-fdr4n to exist\nMachine d0c9p8p4u4g1h6f-nps-420-422-9z6pl-kmv4b: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-78sxml-jbc8sq5wwff6-np-sm-78sxml--managed/providers/Microsoft.Compute/virtualMachines/d0c9p8p4u4g1h6f-nps-420-422-9z6pl-kmv4b to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine d0c9p8p4u4g1h6f-nps-420-422-9z6pl-fdr4n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-78sxml-jbc8sq5wwff6-np-sm-78sxml--managed/providers/Microsoft.Compute/virtualMachines/d0c9p8p4u4g1h6f-nps-420-422-9z6pl-fdr4n to exist\nMachine d0c9p8p4u4g1h6f-nps-420-422-9z6pl-kmv4b: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-78sxml-jbc8sq5wwff6-np-sm-78sxml--managed/providers/Microsoft.Compute/virtualMachines/d0c9p8p4u4g1h6f-nps-420-422-9z6pl-kmv4b to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1bf97134030>: 
    resource group "arm64-vm-cluster-rvxwb88jbj7x": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-rvxwb88jbj7x" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2a654fd8-0468-45a6-8d9a-ee003fb30146
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2a654fd8-0468-45a6-8d9a-ee003fb30146",
      "name": "2a654fd8-0468-45a6-8d9a-ee003fb30146",
      "status": "Failed",
      "startTime": "2026-10-09T11:58:55.836253279Z",
      "endTime": "2026-10-09T12:22:56.194703206Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1bf97134030>: 
    resource group "arm64-vm-cluster-rvxwb88jbj7x": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-rvxwb88jbj7x" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2a654fd8-0468-45a6-8d9a-ee003fb30146
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2a654fd8-0468-45a6-8d9a-ee003fb30146",
      "name": "2a654fd8-0468-45a6-8d9a-ee003fb30146",
      "status": "Failed",
      "startTime": "2026-10-09T11:58:55.836253279Z",
      "endTime": "2026-10-09T12:22:56.194703206Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1d94d9c80a8>: 
    resource group "autoscaling-cluster-tj5rdtq5zbg6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-tj5rdtq5zbg6" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/962a68bc-b321-487b-b276-789b2307a373
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/962a68bc-b321-487b-b276-789b2307a373",
      "name": "962a68bc-b321-487b-b276-789b2307a373",
      "status": "Failed",
      "startTime": "2026-10-09T11:58:48.91579633Z",
      "endTime": "2026-10-09T12:22:56.421671364Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1d94d9c80a8>: 
    resource group "autoscaling-cluster-tj5rdtq5zbg6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-tj5rdtq5zbg6" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/962a68bc-b321-487b-b276-789b2307a373
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/962a68bc-b321-487b-b276-789b2307a373",
      "name": "962a68bc-b321-487b-b276-789b2307a373",
      "status": "Failed",
      "startTime": "2026-10-09T11:58:48.91579633Z",
      "endTime": "2026-10-09T12:22:56.421671364Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x11708fbe8558>: 
    resource group "patch-tags-qgj9df44rmxf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-qgj9df44rmxf" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dae7a46b-a56b-43d2-8533-c8bfc2bffae7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dae7a46b-a56b-43d2-8533-c8bfc2bffae7",
      "name": "dae7a46b-a56b-43d2-8533-c8bfc2bffae7",
      "status": "Failed",
      "startTime": "2026-10-09T11:53:27.886110615Z",
      "endTime": "2026-10-09T12:17:38.071711819Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb8a5khcdtm17dl9aq7kql2ocecgpqo still exists (deletion dispatched at 2026-10-09T11:53:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x11708fbe8558>: 
    resource group "patch-tags-qgj9df44rmxf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-qgj9df44rmxf" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dae7a46b-a56b-43d2-8533-c8bfc2bffae7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dae7a46b-a56b-43d2-8533-c8bfc2bffae7",
      "name": "dae7a46b-a56b-43d2-8533-c8bfc2bffae7",
      "status": "Failed",
      "startTime": "2026-10-09T11:53:27.886110615Z",
      "endTime": "2026-10-09T12:17:38.071711819Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb8a5khcdtm17dl9aq7kql2ocecgpqo still exists (deletion dispatched at 2026-10-09T11:53:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x282f78936210>: 
    resource group "admin-api-breakglass-pwnj2vwmx94x": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-pwnj2vwmx94x" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/efe1d31a-a076-408d-a7e6-045a7416df72
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/efe1d31a-a076-408d-a7e6-045a7416df72",
      "name": "efe1d31a-a076-408d-a7e6-045a7416df72",
      "status": "Failed",
      "startTime": "2026-10-09T11:54:04.049690376Z",
      "endTime": "2026-10-09T12:18:06.189008845Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x282f78936210>: 
    resource group "admin-api-breakglass-pwnj2vwmx94x": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-pwnj2vwmx94x" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/efe1d31a-a076-408d-a7e6-045a7416df72
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/efe1d31a-a076-408d-a7e6-045a7416df72",
      "name": "efe1d31a-a076-408d-a7e6-045a7416df72",
      "status": "Failed",
      "startTime": "2026-10-09T11:54:04.049690376Z",
      "endTime": "2026-10-09T12:18:06.189008845Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x715418a8198>: 
    resource group "rg-hs-pre--d8fntzv77rg5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-6jnw6j" in resourcegroup="rg-hs-pre--d8fntzv77rg5" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f6364b4-168b-4408-93de-ce5f903b20aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f6364b4-168b-4408-93de-ce5f903b20aa",
      "name": "1f6364b4-168b-4408-93de-ce5f903b20aa",
      "status": "Failed",
      "startTime": "2026-10-09T12:05:02.733087351Z",
      "endTime": "2026-10-09T12:29:06.620380667Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb89kg3h59n4o9tks40jdsh5l5jro8g still exists (deletion dispatched at 2026-10-09T12:05:03Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x715418a8198>: 
    resource group "rg-hs-pre--d8fntzv77rg5": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-6jnw6j" in resourcegroup="rg-hs-pre--d8fntzv77rg5" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f6364b4-168b-4408-93de-ce5f903b20aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f6364b4-168b-4408-93de-ce5f903b20aa",
      "name": "1f6364b4-168b-4408-93de-ce5f903b20aa",
      "status": "Failed",
      "startTime": "2026-10-09T12:05:02.733087351Z",
      "endTime": "2026-10-09T12:29:06.620380667Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb89kg3h59n4o9tks40jdsh5l5jro8g still exists (deletion dispatched at 2026-10-09T12:05:03Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3c8ada036198>: 
    resource group "rg-candidate-4-22-7xk4vqr4kmpg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-68b4x4" in resourcegroup="rg-candidate-4-22-7xk4vqr4kmpg" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/25b3de70-92ee-47c8-9155-8c0ca5a96889
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/25b3de70-92ee-47c8-9155-8c0ca5a96889",
      "name": "25b3de70-92ee-47c8-9155-8c0ca5a96889",
      "status": "Failed",
      "startTime": "2026-10-09T12:01:27.302551031Z",
      "endTime": "2026-10-09T12:25:36.409770296Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb89kods8necs5a4oar2jo8ji1411tb still exists (deletion dispatched at 2026-10-09T12:01:28Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3c8ada036198>: 
    resource group "rg-candidate-4-22-7xk4vqr4kmpg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-68b4x4" in resourcegroup="rg-candidate-4-22-7xk4vqr4kmpg" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/25b3de70-92ee-47c8-9155-8c0ca5a96889
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/25b3de70-92ee-47c8-9155-8c0ca5a96889",
      "name": "25b3de70-92ee-47c8-9155-8c0ca5a96889",
      "status": "Failed",
      "startTime": "2026-10-09T12:01:27.302551031Z",
      "endTime": "2026-10-09T12:25:36.409770296Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb89kods8necs5a4oar2jo8ji1411tb still exists (deletion dispatched at 2026-10-09T12:01:28Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x330cf0c08078>: 
    resource group "kusto-logs-dxkfj4jt2n4s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-dxkfj4jt2n4s" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/735a1c9a-7d59-48f3-9927-25e5cda56c30
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/735a1c9a-7d59-48f3-9927-25e5cda56c30",
      "name": "735a1c9a-7d59-48f3-9927-25e5cda56c30",
      "status": "Failed",
      "startTime": "2026-10-09T11:55:18.945459438Z",
      "endTime": "2026-10-09T12:19:26.457578493Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb8a54asvaa4eaak0rehljj280alpei still exists (deletion dispatched at 2026-10-09T11:55:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x330cf0c08078>: 
    resource group "kusto-logs-dxkfj4jt2n4s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-dxkfj4jt2n4s" to finish deleting: GET https://rp.j0395392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/735a1c9a-7d59-48f3-9927-25e5cda56c30
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/735a1c9a-7d59-48f3-9927-25e5cda56c30",
      "name": "735a1c9a-7d59-48f3-9927-25e5cda56c30",
      "status": "Failed",
      "startTime": "2026-10-09T11:55:18.945459438Z",
      "endTime": "2026-10-09T12:19:26.457578493Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb8a54asvaa4eaak0rehljj280alpei still exists (deletion dispatched at 2026-10-09T11:55:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x134adbf680a8>: 
    resource group "rg-cp-ystream-upgrade-4-21-qjqmm9cn4czj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-21-z899h6" in resourcegroup="rg-cp-ystream-upgrade-4-21-qjqmm9cn4czj" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c0cbe6b8-69c0-4d31-bc01-953d33ab1b7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c0cbe6b8-69c0-4d31-bc01-953d33ab1b7b",
      "name": "c0cbe6b8-69c0-4d31-bc01-953d33ab1b7b",
      "status": "Failed",
      "startTime": "2026-10-09T11:56:17.465788464Z",
      "endTime": "2026-10-09T12:20:25.844170504Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x134adbf680a8>: 
    resource group "rg-cp-ystream-upgrade-4-21-qjqmm9cn4czj": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-21-z899h6" in resourcegroup="rg-cp-ystream-upgrade-4-21-qjqmm9cn4czj" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c0cbe6b8-69c0-4d31-bc01-953d33ab1b7b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c0cbe6b8-69c0-4d31-bc01-953d33ab1b7b",
      "name": "c0cbe6b8-69c0-4d31-bc01-953d33ab1b7b",
      "status": "Failed",
      "startTime": "2026-10-09T11:56:17.465788464Z",
      "endTime": "2026-10-09T12:20:25.844170504Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x362e5b21e0c0>: 
    resource group "acr-pull-nb5gr6kq2hj2": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="acr-pull" in resourcegroup="acr-pull-nb5gr6kq2hj2" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/743e94db-c632-4c3e-91d5-7586a4541504
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/743e94db-c632-4c3e-91d5-7586a4541504",
      "name": "743e94db-c632-4c3e-91d5-7586a4541504",
      "status": "Failed",
      "startTime": "2026-10-09T11:59:58.69192494Z",
      "endTime": "2026-10-09T12:24:06.039566807Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb875uo3smk0letllr5okhn9ig34svs still exists (deletion dispatched at 2026-10-09T11:59:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x362e5b21e0c0>: 
    resource group "acr-pull-nb5gr6kq2hj2": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="acr-pull" in resourcegroup="acr-pull-nb5gr6kq2hj2" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/743e94db-c632-4c3e-91d5-7586a4541504
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/743e94db-c632-4c3e-91d5-7586a4541504",
      "name": "743e94db-c632-4c3e-91d5-7586a4541504",
      "status": "Failed",
      "startTime": "2026-10-09T11:59:58.69192494Z",
      "endTime": "2026-10-09T12:24:06.039566807Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb875uo3smk0letllr5okhn9ig34svs still exists (deletion dispatched at 2026-10-09T11:59:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3346ee20e4b0>: 
    resource group "feature-aggregation-vk8bg8lddcq2": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-vk8bg8lddcq2" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4724c843-31f5-4e70-a53a-a9dc39567964
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4724c843-31f5-4e70-a53a-a9dc39567964",
      "name": "4724c843-31f5-4e70-a53a-a9dc39567964",
      "status": "Failed",
      "startTime": "2026-10-09T12:05:11.356025969Z",
      "endTime": "2026-10-09T12:29:15.890678807Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3346ee20e4b0>: 
    resource group "feature-aggregation-vk8bg8lddcq2": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-vk8bg8lddcq2" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4724c843-31f5-4e70-a53a-a9dc39567964
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4724c843-31f5-4e70-a53a-a9dc39567964",
      "name": "4724c843-31f5-4e70-a53a-a9dc39567964",
      "status": "Failed",
      "startTime": "2026-10-09T12:05:11.356025969Z",
      "endTime": "2026-10-09T12:29:15.890678807Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x52f9b8920c0>: 
    resource group "cni-cilium-lqwdfr6cjvht": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-lqwdfr6cjvht" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d5567703-2f10-4c0b-8cf6-70ff35f674c9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d5567703-2f10-4c0b-8cf6-70ff35f674c9",
      "name": "d5567703-2f10-4c0b-8cf6-70ff35f674c9",
      "status": "Failed",
      "startTime": "2026-10-09T11:57:36.843017136Z",
      "endTime": "2026-10-09T12:21:46.074977182Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb86qasjq198a6vq4pknlkm4b3grgda still exists (deletion dispatched at 2026-10-09T11:57:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x52f9b8920c0>: 
    resource group "cni-cilium-lqwdfr6cjvht": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cl" in resourcegroup="cni-cilium-lqwdfr6cjvht" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d5567703-2f10-4c0b-8cf6-70ff35f674c9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d5567703-2f10-4c0b-8cf6-70ff35f674c9",
      "name": "d5567703-2f10-4c0b-8cf6-70ff35f674c9",
      "status": "Failed",
      "startTime": "2026-10-09T11:57:36.843017136Z",
      "endTime": "2026-10-09T12:21:46.074977182Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb86qasjq198a6vq4pknlkm4b3grgda still exists (deletion dispatched at 2026-10-09T11:57:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x19f79d6c0108>: 
    resource group "rg-candidate-4-22-qf28f2r9x98w": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-zdqvrv" in resourcegroup="rg-candidate-4-22-qf28f2r9x98w" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f83f8ddd-649a-4f81-bf26-922da136bfaf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f83f8ddd-649a-4f81-bf26-922da136bfaf",
      "name": "f83f8ddd-649a-4f81-bf26-922da136bfaf",
      "status": "Failed",
      "startTime": "2026-10-09T11:55:22.867180314Z",
      "endTime": "2026-10-09T12:19:26.072146961Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb86qoc4q25fgh3sm80emlj62kju53i still exists (deletion dispatched at 2026-10-09T11:55:23Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x19f79d6c0108>: 
    resource group "rg-candidate-4-22-qf28f2r9x98w": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-zdqvrv" in resourcegroup="rg-candidate-4-22-qf28f2r9x98w" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f83f8ddd-649a-4f81-bf26-922da136bfaf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f83f8ddd-649a-4f81-bf26-922da136bfaf",
      "name": "f83f8ddd-649a-4f81-bf26-922da136bfaf",
      "status": "Failed",
      "startTime": "2026-10-09T11:55:22.867180314Z",
      "endTime": "2026-10-09T12:19:26.072146961Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb86qoc4q25fgh3sm80emlj62kju53i still exists (deletion dispatched at 2026-10-09T11:55:23Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2b3aa8d40060>: 
    resource group "private-keyvault-j49nnk79lk7l": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-j49nnk79lk7l" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0feb53b8-520d-463a-b844-2af1832a77a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0feb53b8-520d-463a-b844-2af1832a77a0",
      "name": "0feb53b8-520d-463a-b844-2af1832a77a0",
      "status": "Failed",
      "startTime": "2026-10-09T11:56:12.658651084Z",
      "endTime": "2026-10-09T12:20:16.061081704Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb87h9gah70vts6mbse5o06n385j3o0 still exists (deletion dispatched at 2026-10-09T11:56:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2b3aa8d40060>: 
    resource group "private-keyvault-j49nnk79lk7l": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-j49nnk79lk7l" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0feb53b8-520d-463a-b844-2af1832a77a0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0feb53b8-520d-463a-b844-2af1832a77a0",
      "name": "0feb53b8-520d-463a-b844-2af1832a77a0",
      "status": "Failed",
      "startTime": "2026-10-09T11:56:12.658651084Z",
      "endTime": "2026-10-09T12:20:16.061081704Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb87h9gah70vts6mbse5o06n385j3o0 still exists (deletion dispatched at 2026-10-09T11:56:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3172298ee060>: 
    resource group "customer-rg-85nx9txxtvjg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-85nx9txxtvjg" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c208ad27-e494-4933-aedb-2b311ea04677
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c208ad27-e494-4933-aedb-2b311ea04677",
      "name": "c208ad27-e494-4933-aedb-2b311ea04677",
      "status": "Failed",
      "startTime": "2026-10-09T11:49:09.861950907Z",
      "endTime": "2026-10-09T12:13:16.248989406Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb87377q9r4c48pr0vofden0ac6k3ii still exists (deletion dispatched at 2026-10-09T11:49:10Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3172298ee060>: 
    resource group "customer-rg-85nx9txxtvjg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-85nx9txxtvjg" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c208ad27-e494-4933-aedb-2b311ea04677
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c208ad27-e494-4933-aedb-2b311ea04677",
      "name": "c208ad27-e494-4933-aedb-2b311ea04677",
      "status": "Failed",
      "startTime": "2026-10-09T11:49:09.861950907Z",
      "endTime": "2026-10-09T12:13:16.248989406Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb87377q9r4c48pr0vofden0ac6k3ii still exists (deletion dispatched at 2026-10-09T11:49:10Z); [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xb001b41e150>: 
    resource group "admin-api-serialconsole-bootdiag-zkmb9sg9qvzk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-zkmb9sg9qvzk" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74bde478-0cee-426a-977f-638be37440d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74bde478-0cee-426a-977f-638be37440d5",
      "name": "74bde478-0cee-426a-977f-638be37440d5",
      "status": "Failed",
      "startTime": "2026-10-09T11:52:47.875676815Z",
      "endTime": "2026-10-09T12:16:55.850520543Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xb001b41e150>: 
    resource group "admin-api-serialconsole-bootdiag-zkmb9sg9qvzk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-zkmb9sg9qvzk" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74bde478-0cee-426a-977f-638be37440d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/74bde478-0cee-426a-977f-638be37440d5",
      "name": "74bde478-0cee-426a-977f-638be37440d5",
      "status": "Failed",
      "startTime": "2026-10-09T11:52:47.875676815Z",
      "endTime": "2026-10-09T12:16:55.850520543Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x10e954cac0c0>: 
    resource group "fips-enabled-g648xvh6nzj9": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-g648xvh6nzj9" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6fa493d6-7fae-4369-b969-8577dd6dfc89
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6fa493d6-7fae-4369-b969-8577dd6dfc89",
      "name": "6fa493d6-7fae-4369-b969-8577dd6dfc89",
      "status": "Failed",
      "startTime": "2026-10-09T11:52:00.540461838Z",
      "endTime": "2026-10-09T12:16:05.840029374Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x10e954cac0c0>: 
    resource group "fips-enabled-g648xvh6nzj9": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="fips-enabled-cluster" in resourcegroup="fips-enabled-g648xvh6nzj9" to finish deleting: GET https://rp.j6085760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6fa493d6-7fae-4369-b969-8577dd6dfc89
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6fa493d6-7fae-4369-b969-8577dd6dfc89",
      "name": "6fa493d6-7fae-4369-b969-8577dd6dfc89",
      "status": "Failed",
      "startTime": "2026-10-09T11:52:00.540461838Z",
      "endTime": "2026-10-09T12:16:05.840029374Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6482688/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j6482688' could not be found."
  }
}
--------------------------------------------------------------------------------

alert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6482688: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6482688/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j6482688' could not be found."
  }
}
--------------------------------------------------------------------------------

alert collection incomplete
failed to fetch svc alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6482688/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
  "error": {
    "code": "ResourceGroupNotFound",
    "message": "Resource group 'hcp-underlay-ci01-j6482688' could not be found."
  }
}
--------------------------------------------------------------------------------

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x18432eff1520>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-kfsdc7" in resourcegroup="rg-np-version-upgrade-kfsdc7-xsdgmhhbvgm4" to finish creating: GET https://rp.j4483328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7ee089c8-2818-4f87-996e-9f68dab666fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7ee089c8-2818-4f87-996e-9f68dab666fd",
      "name": "7ee089c8-2818-4f87-996e-9f68dab666fd",
      "status": "Failed",
      "startTime": "2026-10-09T10:36:13.066941031Z",
      "endTime": "2026-10-09T11:05:15.745369037Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o9h2o2o4l3l4e3q-npupgrade-4-20-x55mz-j2lrd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-kfsdc7-xsdgmhhbvgm4-np-upgrade-kf-f195e85e/providers/Microsoft.Compute/virtualMachines/o9h2o2o4l3l4e3q-npupgrade-4-20-x55mz-j2lrd to exist\nMachine o9h2o2o4l3l4e3q-npupgrade-4-20-x55mz-qw9b7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-kfsdc7-xsdgmhhbvgm4-np-upgrade-kf-f195e85e/providers/Microsoft.Compute/virtualMachines/o9h2o2o4l3l4e3q-npupgrade-4-20-x55mz-qw9b7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o9h2o2o4l3l4e3q-npupgrade-4-20-x55mz-j2lrd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-kfsdc7-xsdgmhhbvgm4-np-upgrade-kf-f195e85e/providers/Microsoft.Compute/virtualMachines/o9h2o2o4l3l4e3q-npupgrade-4-20-x55mz-j2lrd to exist\nMachine o9h2o2o4l3l4e3q-npupgrade-4-20-x55mz-qw9b7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-kfsdc7-xsdgmhhbvgm4-np-upgrade-kf-f195e85e/providers/Microsoft.Compute/virtualMachines/o9h2o2o4l3l4e3q-npupgrade-4-20-x55mz-qw9b7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x18432eff1520>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-kfsdc7" in resourcegroup="rg-np-version-upgrade-kfsdc7-xsdgmhhbvgm4" to finish creating: GET https://rp.j4483328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7ee089c8-2818-4f87-996e-9f68dab666fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7ee089c8-2818-4f87-996e-9f68dab666fd",
      "name": "7ee089c8-2818-4f87-996e-9f68dab666fd",
      "status": "Failed",
      "startTime": "2026-10-09T10:36:13.066941031Z",
      "endTime": "2026-10-09T11:05:15.745369037Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o9h2o2o4l3l4e3q-npupgrade-4-20-x55mz-j2lrd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-kfsdc7-xsdgmhhbvgm4-np-upgrade-kf-f195e85e/providers/Microsoft.Compute/virtualMachines/o9h2o2o4l3l4e3q-npupgrade-4-20-x55mz-j2lrd to exist\nMachine o9h2o2o4l3l4e3q-npupgrade-4-20-x55mz-qw9b7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-kfsdc7-xsdgmhhbvgm4-np-upgrade-kf-f195e85e/providers/Microsoft.Compute/virtualMachines/o9h2o2o4l3l4e3q-npupgrade-4-20-x55mz-qw9b7 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o9h2o2o4l3l4e3q-npupgrade-4-20-x55mz-j2lrd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-kfsdc7-xsdgmhhbvgm4-np-upgrade-kf-f195e85e/providers/Microsoft.Compute/virtualMachines/o9h2o2o4l3l4e3q-npupgrade-4-20-x55mz-j2lrd to exist\nMachine o9h2o2o4l3l4e3q-npupgrade-4-20-x55mz-qw9b7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-kfsdc7-xsdgmhhbvgm4-np-upgrade-kf-f195e85e/providers/Microsoft.Compute/virtualMachines/o9h2o2o4l3l4e3q-npupgrade-4-20-x55mz-qw9b7 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x16e23cafa520>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-cwpvfwwq89gb" to finish creating: GET https://rp.j4483328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e7b7c93c-6eac-40c9-80bd-d8fc4f7fb529
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e7b7c93c-6eac-40c9-80bd-d8fc4f7fb529",
      "name": "e7b7c93c-6eac-40c9-80bd-d8fc4f7fb529",
      "status": "Failed",
      "startTime": "2026-10-09T10:23:47.421215993Z",
      "endTime": "2026-10-09T10:52:51.928906707Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c8m0p9k9w9k3x0r-np-1-xvxq5-sdcw7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-cwpvfwwq89gb--managed/providers/Microsoft.Compute/virtualMachines/c8m0p9k9w9k3x0r-np-1-xvxq5-sdcw7 to exist\nMachine c8m0p9k9w9k3x0r-np-1-xvxq5-xf82w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-cwpvfwwq89gb--managed/providers/Microsoft.Compute/virtualMachines/c8m0p9k9w9k3x0r-np-1-xvxq5-xf82w to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c8m0p9k9w9k3x0r-np-1-xvxq5-sdcw7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-cwpvfwwq89gb--managed/providers/Microsoft.Compute/virtualMachines/c8m0p9k9w9k3x0r-np-1-xvxq5-sdcw7 to exist\nMachine c8m0p9k9w9k3x0r-np-1-xvxq5-xf82w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-cwpvfwwq89gb--managed/providers/Microsoft.Compute/virtualMachines/c8m0p9k9w9k3x0r-np-1-xvxq5-xf82w to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:138]: failed to create node pool "np-1" for private keyvault cluster "private-kv-cluster"
Unexpected error:
    <*fmt.wrapError | 0x16e23cafa520>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-kv-cluster" in resourcegroup="private-keyvault-cwpvfwwq89gb" to finish creating: GET https://rp.j4483328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e7b7c93c-6eac-40c9-80bd-d8fc4f7fb529
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e7b7c93c-6eac-40c9-80bd-d8fc4f7fb529",
      "name": "e7b7c93c-6eac-40c9-80bd-d8fc4f7fb529",
      "status": "Failed",
      "startTime": "2026-10-09T10:23:47.421215993Z",
      "endTime": "2026-10-09T10:52:51.928906707Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine c8m0p9k9w9k3x0r-np-1-xvxq5-sdcw7: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-cwpvfwwq89gb--managed/providers/Microsoft.Compute/virtualMachines/c8m0p9k9w9k3x0r-np-1-xvxq5-sdcw7 to exist\nMachine c8m0p9k9w9k3x0r-np-1-xvxq5-xf82w: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-cwpvfwwq89gb--managed/providers/Microsoft.Compute/virtualMachines/c8m0p9k9w9k3x0r-np-1-xvxq5-xf82w to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine c8m0p9k9w9k3x0r-np-1-xvxq5-sdcw7: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-cwpvfwwq89gb--managed/providers/Microsoft.Compute/virtualMachines/c8m0p9k9w9k3x0r-np-1-xvxq5-sdcw7 to exist\nMachine c8m0p9k9w9k3x0r-np-1-xvxq5-xf82w: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-cwpvfwwq89gb--managed/providers/Microsoft.Compute/virtualMachines/c8m0p9k9w9k3x0r-np-1-xvxq5-xf82w to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T10:08:14Z
  Ended:       2026-10-09T10:15:16Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j4483328-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: More than 5% of frontend requests for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j4483328-svc).
Firing 2:
  State:       Resolved
  Started:     2026-10-09T10:31:13Z
  Ended:       2026-10-09T10:40:15Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j4483328-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: More than 5% of frontend requests for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j4483328-svc).

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x126a68f88048>: 
    resource group "nodepool-update-nodes-ppzw4fscb65q": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-ppzw4fscb65q" to finish deleting: GET https://rp.j7543168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5402a6f6-777b-405f-bf42-17f6202e880d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5402a6f6-777b-405f-bf42-17f6202e880d",
      "name": "5402a6f6-777b-405f-bf42-17f6202e880d",
      "status": "Failed",
      "startTime": "2026-10-09T09:00:29.162024664Z",
      "endTime": "2026-10-09T09:24:36.108080611Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb5gs6ackm9riurrcuq3gadk7un22q7 still exists (deletion dispatched at 2026-10-09T09:00:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x126a68f88048>: 
    resource group "nodepool-update-nodes-ppzw4fscb65q": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-ppzw4fscb65q" to finish deleting: GET https://rp.j7543168.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5402a6f6-777b-405f-bf42-17f6202e880d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5402a6f6-777b-405f-bf42-17f6202e880d",
      "name": "5402a6f6-777b-405f-bf42-17f6202e880d",
      "status": "Failed",
      "startTime": "2026-10-09T09:00:29.162024664Z",
      "endTime": "2026-10-09T09:24:36.108080611Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb5gs6ackm9riurrcuq3gadk7un22q7 still exists (deletion dispatched at 2026-10-09T09:00:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x95c6dd09a20>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-zsv74hnjtfn5": PUT https://rp.j2592384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zsv74hnjtfn5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:102]: failed to create workers for SWIFT outbound controls
Unexpected error:
    <*fmt.wrapError | 0x95c6dd09a20>: 
    failed to create NodePool engineering-workers: failed starting nodepool creation "engineering-workers" for cluster "engineering-hcp-cluster" in resourcegroup="kusto-logs-zsv74hnjtfn5": PUT https://rp.j2592384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-zsv74hnjtfn5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster/nodePools/engineering-workers
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "2 errors occurred",
        "details": [
          {
            "code": "InvalidRequestContent",
            "message": "Too long: may not be more than 15 bytes",
            "target": "id"
          },
          {
            "code": "InvalidRequestContent",
            "message": "Invalid value: \"engineering-workers\": (must be a valid DNS RFC 1035 label)",
            "target": "id"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0x128ebe5153a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-nnh2p9v2fmnj" to finish creating: GET https://rp.j2592384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f161e6c5-fc32-45c6-9cfa-cf016a200caf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f161e6c5-fc32-45c6-9cfa-cf016a200caf",
      "name": "f161e6c5-fc32-45c6-9cfa-cf016a200caf",
      "status": "Failed",
      "startTime": "2026-10-09T08:16:08.568494735Z",
      "endTime": "2026-10-09T08:45:11.046514818Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine private-ingress-np-1-q6g8p-kt5pq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-nnh2p9v2fmnj--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-q6g8p-kt5pq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-q6g8p-kt5pq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-nnh2p9v2fmnj--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-q6g8p-kt5pq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:123]: failed to create node pool "np-1" for private ingress cluster "private-ingress"
Unexpected error:
    <*fmt.wrapError | 0x128ebe5153a0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "private-ingress" in resourcegroup="private-ingress-nnh2p9v2fmnj" to finish creating: GET https://rp.j2592384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f161e6c5-fc32-45c6-9cfa-cf016a200caf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f161e6c5-fc32-45c6-9cfa-cf016a200caf",
      "name": "f161e6c5-fc32-45c6-9cfa-cf016a200caf",
      "status": "Failed",
      "startTime": "2026-10-09T08:16:08.568494735Z",
      "endTime": "2026-10-09T08:45:11.046514818Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine private-ingress-np-1-q6g8p-kt5pq: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-nnh2p9v2fmnj--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-q6g8p-kt5pq to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine private-ingress-np-1-q6g8p-kt5pq: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-nnh2p9v2fmnj--managed/providers/Microsoft.Compute/virtualMachines/private-ingress-np-1-q6g8p-kt5pq to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T08:36:10Z
  Ended:       2026-10-09T08:43:12Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2592384-svc", component="backend", name="actualhostedcluster", region="westus3", severity="warning"
  Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-j492kp2w2kt8/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-j492kp2w2kt8/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
    <armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
    <armredhatopenshifthcp.ProvisioningState>: Failed

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-w8jdcn"
Unexpected error:
    <*fmt.wrapError | 0xd9c285fc740>: 
    DNS for route host agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0xd9c285fc6e0>{
            msg: "DNS for agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-20-w8jdcn"
Unexpected error:
    <*fmt.wrapError | 0xd9c285fc740>: 
    DNS for route host agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    {
        msg: "DNS for route host agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
        err: <*fmt.wrapError | 0xd9c285fc6e0>{
            msg: "DNS for agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-nbzfm.apps.aro.s6b6j4f3z0n0v5e.3wbk.j0006144.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded",
            err: <context.deadlineExceededError>{},
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:495]: failed to create node pool
Unexpected error:
    <*fmt.wrapError | 0x19713c8ce520>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "acr-pull" in resourcegroup="acr-pull-dfz6ss2tmqmt" to finish creating: GET https://rp.j0006144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5c0e150d-aecf-4c68-b9dd-c7ae7dbf7400
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5c0e150d-aecf-4c68-b9dd-c7ae7dbf7400",
      "name": "5c0e150d-aecf-4c68-b9dd-c7ae7dbf7400",
      "status": "Failed",
      "startTime": "2026-10-09T08:12:53.421461459Z",
      "endTime": "2026-10-09T08:42:02.147015674Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:495]: failed to create node pool
Unexpected error:
    <*fmt.wrapError | 0x19713c8ce520>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "acr-pull" in resourcegroup="acr-pull-dfz6ss2tmqmt" to finish creating: GET https://rp.j0006144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5c0e150d-aecf-4c68-b9dd-c7ae7dbf7400
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5c0e150d-aecf-4c68-b9dd-c7ae7dbf7400",
      "name": "5c0e150d-aecf-4c68-b9dd-c7ae7dbf7400",
      "status": "Failed",
      "startTime": "2026-10-09T08:12:53.421461459Z",
      "endTime": "2026-10-09T08:42:02.147015674Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T08:14:32Z
  Ended:       2026-10-09T08:32:31Z
  Severity:    Sev3
  Labels:      alertname="FleetControllerRetryHotLoop", cluster="ci01-j0006144-svc", component="fleet", name="nodepoolcontroller", region="westus3", severity="warning"
  Description: Fleet controller workqueue nodepoolcontroller has a retry ratio of > 50% sustained over 10 minutes, indicating most queue activity is failed retries rather than fresh work.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.008s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xba7a1123110>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.008s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xba7a1123110>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T07:28:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0934272-mgmt-1", component="kubernetes-infrastructure", container="controller", endpoint="http", environment="ci01", instance="10.128.64.212:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0934272/providers/microsoft.monitor/accounts/hcps-j0934272", namespace="ocm-arohcpci01-2tb42n0fgkuoc1sge8efonl494ts6f8p-q5w0c9a9b5r6x3n", pod="cloud-network-config-controller-5cf98f8c95-5bd8j", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a135b966-4dab-47fb-8cc2-d75dfe7326f1"
  Description: Pod ocm-arohcpci01-2tb42n0fgkuoc1sge8efonl494ts6f8p-q5w0c9a9b5r6x3n/cloud-network-config-controller-5cf98f8c95-5bd8j (controller) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T07:39:59Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0934272-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.212:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0934272/providers/microsoft.monitor/accounts/hcps-j0934272", namespace="ocm-arohcpci01-2tb42n0fgkuoc1sge8efonl494ts6f8p-q5w0c9a9b5r6x3n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qf8xh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tb42n0fgkuoc1sge8efonl494ts6f8p-q5w0c9a9b5r6x3n/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T07:39:59Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0934272-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.212:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0934272/providers/microsoft.monitor/accounts/hcps-j0934272", namespace="ocm-arohcpci01-2tb42n0fgkuoc1sge8efonl494ts6f8p-q5w0c9a9b5r6x3n", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qf8xh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tb42n0fgkuoc1sge8efonl494ts6f8p-q5w0c9a9b5r6x3n/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2f3c0aa90108>: 
    resource group "admin-credential-lifecycle-test-lxkxdsm95v5q": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-xck96d" in resourcegroup="admin-credential-lifecycle-test-lxkxdsm95v5q" to finish deleting: GET https://rp.j4643712.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d7634497-b76b-417d-ba61-d17625706ddd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d7634497-b76b-417d-ba61-d17625706ddd",
      "name": "d7634497-b76b-417d-ba61-d17625706ddd",
      "status": "Failed",
      "startTime": "2026-10-09T05:12:01.285111592Z",
      "endTime": "2026-10-09T05:36:09.092898944Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb2dcbc75fd0t83s3l86makdqbo52ni still exists (deletion dispatched at 2026-10-09T05:12:02Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2f3c0aa90108>: 
    resource group "admin-credential-lifecycle-test-lxkxdsm95v5q": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-xck96d" in resourcegroup="admin-credential-lifecycle-test-lxkxdsm95v5q" to finish deleting: GET https://rp.j4643712.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d7634497-b76b-417d-ba61-d17625706ddd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d7634497-b76b-417d-ba61-d17625706ddd",
      "name": "d7634497-b76b-417d-ba61-d17625706ddd",
      "status": "Failed",
      "startTime": "2026-10-09T05:12:01.285111592Z",
      "endTime": "2026-10-09T05:36:09.092898944Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb2dcbc75fd0t83s3l86makdqbo52ni still exists (deletion dispatched at 2026-10-09T05:12:02Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (2)"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T05:55:14Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4643712-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.200:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4643712/providers/microsoft.monitor/accounts/hcps-j4643712", namespace="ocm-arohcpci01-2tb2dcbc75fd0t83s3l86makdqbo52ni-j9n9r9c9m0a4s7v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-4rlvg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tb2dcbc75fd0t83s3l86makdqbo52ni-j9n9r9c9m0a4s7v/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-10-09T05:55:14Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4643712-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.200:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4643712/providers/microsoft.monitor/accounts/hcps-j4643712", namespace="ocm-arohcpci01-2tb2dcbc75fd0t83s3l86makdqbo52ni-j9n9r9c9m0a4s7v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-4rlvg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tb2dcbc75fd0t83s3l86makdqbo52ni-j9n9r9c9m0a4s7v/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-10-09T06:09:10Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4643712-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.200:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4643712/providers/microsoft.monitor/accounts/hcps-j4643712", namespace="ocm-arohcpci01-2tb2dcbc75fd0t83s3l86makdqbo52ni-j9n9r9c9m0a4s7v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-4rlvg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tb2dcbc75fd0t83s3l86makdqbo52ni-j9n9r9c9m0a4s7v/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-10-09T06:09:10Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4643712-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.64.200:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4643712/providers/microsoft.monitor/accounts/hcps-j4643712", namespace="ocm-arohcpci01-2tb2dcbc75fd0t83s3l86makdqbo52ni-j9n9r9c9m0a4s7v", pod="arohcp-monitor-kube-state-metrics-7988dc648c-4rlvg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2tb2dcbc75fd0t83s3l86makdqbo52ni-j9n9r9c9m0a4s7v/cluster-api has not matched the expected number of replicas for longer than 30 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-4-21-cxd2vj"
Unexpected error:
    <*fmt.wrapError | 0x3b58d6261e60>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-4-21-cxd2vj" in resourcegroup="rg-cp-ystream-deny-pods-4-21-hw27qjjdkxw6" to finish creating: GET https://rp.j8469504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77583a7d-a965-4558-a265-ffd7e06f41b9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77583a7d-a965-4558-a265-ffd7e06f41b9",
      "name": "77583a7d-a965-4558-a265-ffd7e06f41b9",
      "status": "Failed",
      "startTime": "2026-10-09T04:44:07.077624496Z",
      "endTime": "2026-10-09T05:13:07.452067539Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a9a1f6e4r7f8b6e-np-deny-pods-j2b5z-btsbp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-4-21-hw27qjjdkxw6-cp-ystream-de-81b4ef58/providers/Microsoft.Compute/virtualMachines/a9a1f6e4r7f8b6e-np-deny-pods-j2b5z-btsbp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a9a1f6e4r7f8b6e-np-deny-pods-j2b5z-btsbp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-4-21-hw27qjjdkxw6-cp-ystream-de-81b4ef58/providers/Microsoft.Compute/virtualMachines/a9a1f6e4r7f8b6e-np-deny-pods-j2b5z-btsbp to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_blocked_pod_creation.go:201]: failed to create node pool "np-deny-pods" for cluster "cp-ystream-deny-pods-4-21-cxd2vj"
Unexpected error:
    <*fmt.wrapError | 0x3b58d6261e60>: 
    failed to create NodePool np-deny-pods: failed waiting for nodepool="np-deny-pods" for cluster "cp-ystream-deny-pods-4-21-cxd2vj" in resourcegroup="rg-cp-ystream-deny-pods-4-21-hw27qjjdkxw6" to finish creating: GET https://rp.j8469504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77583a7d-a965-4558-a265-ffd7e06f41b9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77583a7d-a965-4558-a265-ffd7e06f41b9",
      "name": "77583a7d-a965-4558-a265-ffd7e06f41b9",
      "status": "Failed",
      "startTime": "2026-10-09T04:44:07.077624496Z",
      "endTime": "2026-10-09T05:13:07.452067539Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine a9a1f6e4r7f8b6e-np-deny-pods-j2b5z-btsbp: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-4-21-hw27qjjdkxw6-cp-ystream-de-81b4ef58/providers/Microsoft.Compute/virtualMachines/a9a1f6e4r7f8b6e-np-deny-pods-j2b5z-btsbp to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine a9a1f6e4r7f8b6e-np-deny-pods-j2b5z-btsbp: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-deny-pods-4-21-hw27qjjdkxw6-cp-ystream-de-81b4ef58/providers/Microsoft.Compute/virtualMachines/a9a1f6e4r7f8b6e-np-deny-pods-j2b5z-btsbp to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

time=2026-10-09T03:02:25.127Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n  Kind: ARM\n  Template: templates/svc-cluster.bicep\n  Parameters: configurations/svc-cluster.tmpl.bicepparam"
time=2026-10-09T03:02:27.026Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster
time=2026-10-09T03:02:29.104Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=9100c383496433b8981e04307d4ff94d8b8595203688da3bb283a6fce4e49fea portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9299456-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9100c383496433b8981e04307d4ff94d8b8595203688da3bb283a6fce4e49fea
time=2026-10-09T03:32:24.844Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-10-09T03:29:55Z
  Severity:    Sev3
  Labels:      alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j9299456-mgmt-2", component="monitoring-infrastructure", region="westus3", severity="3"
  Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing.

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xd266b380078>: 
    resource group "patch-tags-6wvvmpdg8f9c": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-6wvvmpdg8f9c" to finish deleting: GET https://rp.j4430336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1803d96-bd6c-40cd-936a-45f1f9c9c199
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1803d96-bd6c-40cd-936a-45f1f9c9c199",
      "name": "f1803d96-bd6c-40cd-936a-45f1f9c9c199",
      "status": "Failed",
      "startTime": "2026-10-09T03:10:26.245027108Z",
      "endTime": "2026-10-09T03:34:35.549070453Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb0kpdkn33trftvfg3ohnmt2hf90ta7 still exists (deletion dispatched at 2026-10-09T03:10:26Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xd266b380078>: 
    resource group "patch-tags-6wvvmpdg8f9c": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-6wvvmpdg8f9c" to finish deleting: GET https://rp.j4430336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1803d96-bd6c-40cd-936a-45f1f9c9c199
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f1803d96-bd6c-40cd-936a-45f1f9c9c199",
      "name": "f1803d96-bd6c-40cd-936a-45f1f9c9c199",
      "status": "Failed",
      "startTime": "2026-10-09T03:10:26.245027108Z",
      "endTime": "2026-10-09T03:34:35.549070453Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb0kpdkn33trftvfg3ohnmt2hf90ta7 still exists (deletion dispatched at 2026-10-09T03:10:26Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x2741419abce0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:388]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0x2741419abce0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    {
        s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist",
    }

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2bebdca64108>: 
    resource group "customer-rg-n6b8nxvh5gzl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-n6b8nxvh5gzl" to finish deleting: GET https://rp.j4430336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b47eee2-4558-47ec-876a-e910a64edcab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b47eee2-4558-47ec-876a-e910a64edcab",
      "name": "0b47eee2-4558-47ec-876a-e910a64edcab",
      "status": "Failed",
      "startTime": "2026-10-09T03:09:52.911264338Z",
      "endTime": "2026-10-09T03:33:54.399258177Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x2bebdca64108>: 
    resource group "customer-rg-n6b8nxvh5gzl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-n6b8nxvh5gzl" to finish deleting: GET https://rp.j4430336.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b47eee2-4558-47ec-876a-e910a64edcab
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0b47eee2-4558-47ec-876a-e910a64edcab",
      "name": "0b47eee2-4558-47ec-876a-e910a64edcab",
      "status": "Failed",
      "startTime": "2026-10-09T03:09:52.911264338Z",
      "endTime": "2026-10-09T03:33:54.399258177Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullc5dl28vw558j.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x34f9246db800>: 
    VerifyImagePulled(acrpullc5dl28vw558j.azurecr.io/ubi-minimal) timed out after 2m0.002s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(acrpullc5dl28vw558j.azurecr.io/ubi-minimal) timed out after 2m0.002s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x34f9246db6e0>{
            msg: "failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x34f9246db5e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullc5dl28vw558j.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x34f9246db800>: 
    VerifyImagePulled(acrpullc5dl28vw558j.azurecr.io/ubi-minimal) timed out after 2m0.002s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(acrpullc5dl28vw558j.azurecr.io/ubi-minimal) timed out after 2m0.002s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x34f9246db6e0>{
            msg: "failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x34f9246db5e0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x19485d7e23a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-h78g452l2k2m" to finish creating: GET https://rp.j1850240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2445641e-62af-46ed-8294-3ebf3cab568c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2445641e-62af-46ed-8294-3ebf3cab568c",
      "name": "2445641e-62af-46ed-8294-3ebf3cab568c",
      "status": "Failed",
      "startTime": "2026-10-09T03:36:09.491861222Z",
      "endTime": "2026-10-09T04:05:13.503264505Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-h78g452l2k2m--managed/providers/Microsoft.Compute/virtualMachines/t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-h78g452l2k2m--managed/providers/Microsoft.Compute/virtualMachines/t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0x19485d7e23a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-h78g452l2k2m" to finish creating: GET https://rp.j1850240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2445641e-62af-46ed-8294-3ebf3cab568c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2445641e-62af-46ed-8294-3ebf3cab568c",
      "name": "2445641e-62af-46ed-8294-3ebf3cab568c",
      "status": "Failed",
      "startTime": "2026-10-09T03:36:09.491861222Z",
      "endTime": "2026-10-09T04:05:13.503264505Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-h78g452l2k2m--managed/providers/Microsoft.Compute/virtualMachines/t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-h78g452l2k2m--managed/providers/Microsoft.Compute/virtualMachines/t9g4t9s9x3s5g2e-np-delete-main-wlzrv-h29dk to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x239dd5bc9700>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-sdfdh6slg4p7" to finish creating: GET https://rp.j7240576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd44afef-8e0c-422b-a28f-1a1fdf5406e3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd44afef-8e0c-422b-a28f-1a1fdf5406e3",
      "name": "cd44afef-8e0c-422b-a28f-1a1fdf5406e3",
      "status": "Failed",
      "startTime": "2026-10-09T03:04:16.687507434Z",
      "endTime": "2026-10-09T03:33:25.201409198Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q7c4s8i0s9p3s5q-np-1-bvbvq-h94h9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdfdh6slg4p7--managed/providers/Microsoft.Compute/virtualMachines/q7c4s8i0s9p3s5q-np-1-bvbvq-h94h9 to exist\nMachine q7c4s8i0s9p3s5q-np-1-bvbvq-vjj64: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdfdh6slg4p7--managed/providers/Microsoft.Compute/virtualMachines/q7c4s8i0s9p3s5q-np-1-bvbvq-vjj64 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q7c4s8i0s9p3s5q-np-1-bvbvq-h94h9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdfdh6slg4p7--managed/providers/Microsoft.Compute/virtualMachines/q7c4s8i0s9p3s5q-np-1-bvbvq-h94h9 to exist\nMachine q7c4s8i0s9p3s5q-np-1-bvbvq-vjj64: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdfdh6slg4p7--managed/providers/Microsoft.Compute/virtualMachines/q7c4s8i0s9p3s5q-np-1-bvbvq-vjj64 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0x239dd5bc9700>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-sdfdh6slg4p7" to finish creating: GET https://rp.j7240576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd44afef-8e0c-422b-a28f-1a1fdf5406e3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd44afef-8e0c-422b-a28f-1a1fdf5406e3",
      "name": "cd44afef-8e0c-422b-a28f-1a1fdf5406e3",
      "status": "Failed",
      "startTime": "2026-10-09T03:04:16.687507434Z",
      "endTime": "2026-10-09T03:33:25.201409198Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine q7c4s8i0s9p3s5q-np-1-bvbvq-h94h9: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdfdh6slg4p7--managed/providers/Microsoft.Compute/virtualMachines/q7c4s8i0s9p3s5q-np-1-bvbvq-h94h9 to exist\nMachine q7c4s8i0s9p3s5q-np-1-bvbvq-vjj64: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdfdh6slg4p7--managed/providers/Microsoft.Compute/virtualMachines/q7c4s8i0s9p3s5q-np-1-bvbvq-vjj64 to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine q7c4s8i0s9p3s5q-np-1-bvbvq-h94h9: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdfdh6slg4p7--managed/providers/Microsoft.Compute/virtualMachines/q7c4s8i0s9p3s5q-np-1-bvbvq-h94h9 to exist\nMachine q7c4s8i0s9p3s5q-np-1-bvbvq-vjj64: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-sdfdh6slg4p7--managed/providers/Microsoft.Compute/virtualMachines/q7c4s8i0s9p3s5q-np-1-bvbvq-vjj64 to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x4a16d2b40f0>: 
    resource group "rg-candidate-5-1-trdpp8hx8tpd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-1-jjf49v" in resourcegroup="rg-candidate-5-1-trdpp8hx8tpd" to finish deleting: GET https://rp.j1851392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/292a5ec0-f28b-48a4-89bc-34c6f69d9a46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/292a5ec0-f28b-48a4-89bc-34c6f69d9a46",
      "name": "292a5ec0-f28b-48a4-89bc-34c6f69d9a46",
      "status": "Failed",
      "startTime": "2026-10-09T03:05:28.202107106Z",
      "endTime": "2026-10-09T03:29:29.34914283Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb0dh1i1h74jbtnnmtcppkg9ambqd8h still exists (deletion dispatched at 2026-10-09T03:05:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x4a16d2b40f0>: 
    resource group "rg-candidate-5-1-trdpp8hx8tpd": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-1-jjf49v" in resourcegroup="rg-candidate-5-1-trdpp8hx8tpd" to finish deleting: GET https://rp.j1851392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/292a5ec0-f28b-48a4-89bc-34c6f69d9a46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/292a5ec0-f28b-48a4-89bc-34c6f69d9a46",
      "name": "292a5ec0-f28b-48a4-89bc-34c6f69d9a46",
      "status": "Failed",
      "startTime": "2026-10-09T03:05:28.202107106Z",
      "endTime": "2026-10-09T03:29:29.34914283Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb0dh1i1h74jbtnnmtcppkg9ambqd8h still exists (deletion dispatched at 2026-10-09T03:05:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1086ab296168>: 
    resource group "v20261001preview-cr9lplpfwvph": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-cr9lplpfwvph" to finish deleting: GET https://rp.j1851392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/10d85540-0575-4e0a-abac-3407d1548f97
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/10d85540-0575-4e0a-abac-3407d1548f97",
      "name": "10d85540-0575-4e0a-abac-3407d1548f97",
      "status": "Failed",
      "startTime": "2026-10-09T03:04:56.263351094Z",
      "endTime": "2026-10-09T03:28:59.770735963Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb0dh6ok3afvcebiteja6v1aqvho0u6 still exists (deletion dispatched at 2026-10-09T03:04:56Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x1086ab296168>: 
    resource group "v20261001preview-cr9lplpfwvph": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20261001" in resourcegroup="v20261001preview-cr9lplpfwvph" to finish deleting: GET https://rp.j1851392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/10d85540-0575-4e0a-abac-3407d1548f97
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/10d85540-0575-4e0a-abac-3407d1548f97",
      "name": "10d85540-0575-4e0a-abac-3407d1548f97",
      "status": "Failed",
      "startTime": "2026-10-09T03:04:56.263351094Z",
      "endTime": "2026-10-09T03:28:59.770735963Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb0dh6ok3afvcebiteja6v1aqvho0u6 still exists (deletion dispatched at 2026-10-09T03:04:56Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3e5591194198>: 
    resource group "rg-hs-pre--zqt4j66rqwpg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-84qkld" in resourcegroup="rg-hs-pre--zqt4j66rqwpg" to finish deleting: GET https://rp.j1851392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aabc61f6-8a67-488f-a48c-a579beb22f14
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aabc61f6-8a67-488f-a48c-a579beb22f14",
      "name": "aabc61f6-8a67-488f-a48c-a579beb22f14",
      "status": "Failed",
      "startTime": "2026-10-09T03:02:21.534993847Z",
      "endTime": "2026-10-09T03:26:29.648167314Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb0dk6mkvl6568flpu0m9h3go953hh5 still exists (deletion dispatched at 2026-10-09T03:02:21Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x3e5591194198>: 
    resource group "rg-hs-pre--zqt4j66rqwpg": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-84qkld" in resourcegroup="rg-hs-pre--zqt4j66rqwpg" to finish deleting: GET https://rp.j1851392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aabc61f6-8a67-488f-a48c-a579beb22f14
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aabc61f6-8a67-488f-a48c-a579beb22f14",
      "name": "aabc61f6-8a67-488f-a48c-a579beb22f14",
      "status": "Failed",
      "startTime": "2026-10-09T03:02:21.534993847Z",
      "endTime": "2026-10-09T03:26:29.648167314Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tb0dk6mkvl6568flpu0m9h3go953hh5 still exists (deletion dispatched at 2026-10-09T03:02:21Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:456]: error for disabled boot diagnostics should contain 409 status code
Expected
    <string>: failed to send request: Get "https://admin.j1851392.hcpsvc.osadev.cloud/admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-serialconsole-bootdiag-mx7j6swwr25v/providers/Microsoft.RedHatOpenshift/hcpOpenShiftClusters/sre-hcp-cluster-bootdiag/serialconsole?vmName=d3i6t7c9q4h0k8j-worker-wrlpp-npg85": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
to contain substring
    <string>: 409
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:456]: error for disabled boot diagnostics should contain 409 status code
Expected
    <string>: failed to send request: Get "https://admin.j1851392.hcpsvc.osadev.cloud/admin/v1/hcp/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-api-serialconsole-bootdiag-mx7j6swwr25v/providers/Microsoft.RedHatOpenshift/hcpOpenShiftClusters/sre-hcp-cluster-bootdiag/serialconsole?vmName=d3i6t7c9q4h0k8j-worker-wrlpp-npg85": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
to contain substring
    <string>: 409

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc360c750060>: 
    resource group "rg-cp-ystream-upgrade-4-22-gx4vltwqcd64": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-pgmlq6" in resourcegroup="rg-cp-ystream-upgrade-4-22-gx4vltwqcd64" to finish deleting: GET https://rp.j1851392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ddf87703-cd9c-4434-ab6b-2db7a064f4fc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ddf87703-cd9c-4434-ab6b-2db7a064f4fc",
      "name": "ddf87703-cd9c-4434-ab6b-2db7a064f4fc",
      "status": "Failed",
      "startTime": "2026-10-09T03:09:13.799569988Z",
      "endTime": "2026-10-09T03:33:19.091161643Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc360c750060>: 
    resource group "rg-cp-ystream-upgrade-4-22-gx4vltwqcd64": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-pgmlq6" in resourcegroup="rg-cp-ystream-upgrade-4-22-gx4vltwqcd64" to finish deleting: GET https://rp.j1851392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ddf87703-cd9c-4434-ab6b-2db7a064f4fc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ddf87703-cd9c-4434-ab6b-2db7a064f4fc",
      "name": "ddf87703-cd9c-4434-ab6b-2db7a064f4fc",
      "status": "Failed",
      "startTime": "2026-10-09T03:09:13.799569988Z",
      "endTime": "2026-10-09T03:33:19.091161643Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x124d3091c198>: 
    resource group "rg-candidate-5-0-wjcmd5gpjktw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-q7hvjk" in resourcegroup="rg-candidate-5-0-wjcmd5gpjktw" to finish deleting: GET https://rp.j1851392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/323be21f-0ba0-4b77-9aca-6bab469f3659
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/323be21f-0ba0-4b77-9aca-6bab469f3659",
      "name": "323be21f-0ba0-4b77-9aca-6bab469f3659",
      "status": "Failed",
      "startTime": "2026-10-09T03:05:03.476461089Z",
      "endTime": "2026-10-09T03:29:09.085409713Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0x124d3091c198>: 
    resource group "rg-candidate-5-0-wjcmd5gpjktw": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-q7hvjk" in resourcegroup="rg-candidate-5-0-wjcmd5gpjktw" to finish deleting: GET https://rp.j1851392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/323be21f-0ba0-4b77-9aca-6bab469f3659
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/323be21f-0ba0-4b77-9aca-6bab469f3659",
      "name": "323be21f-0ba0-4b77-9aca-6bab469f3659",
      "status": "Failed",
      "startTime": "2026-10-09T03:05:03.476461089Z",
      "endTime": "2026-10-09T03:29:09.085409713Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullctpjh66q2dr2.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x97759ef0500>: 
    VerifyImagePulled(acrpullctpjh66q2dr2.azurecr.io/ubi-minimal) timed out after 2m0.001s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(acrpullctpjh66q2dr2.azurecr.io/ubi-minimal) timed out after 2m0.001s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x97759ef04a0>{
            msg: "failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x97759f21ec0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_container_registry.go:257]: [day 2] pod in namespace acr-pull-test-day2 never pulled image ubi-minimal from private ACR acrpullctpjh66q2dr2.azurecr.io
Unexpected error:
    <*fmt.wrapError | 0x97759ef0500>: 
    VerifyImagePulled(acrpullctpjh66q2dr2.azurecr.io/ubi-minimal) timed out after 2m0.001s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded
    {
        msg: "VerifyImagePulled(acrpullctpjh66q2dr2.azurecr.io/ubi-minimal) timed out after 2m0.001s: failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
        err: <*fmt.wrapError | 0x97759ef04a0>{
            msg: "failed to list pods in namespace acr-pull-test-day2: client rate limiter Wait returned an error: context deadline exceeded",
            err: <*fmt.wrapError | 0x97759f21ec0>{
                msg: "client rate limiter Wait returned an error: context deadline exceeded",
                err: <context.deadlineExceededError>{},
            },
        },
    }
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-10-09T02:59:09Z
  Ended:       2026-10-09T03:12:06Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1851392-svc", component="backend", name="clusterresources", region="westus3", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x284ab7439230>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-4qsn8gb8p28c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x284ab7439230>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-4qsn8gb8p28c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x2a25db7081b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-gkw6wzhpx84w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x2a25db7081b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-gkw6wzhpx84w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x3f756cdcc8d0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-qq6g9wz95bjw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x3f756cdcc8d0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-qq6g9wz95bjw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x30713b85be60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-tffs96fmr5p6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x30713b85be60>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-tffs96fmr5p6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x25bb28101bc0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-bbmhspx64hh7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x25bb28101bc0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-bbmhspx64hh7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x15ccc8e3e260>: 
    failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-s4lpkb" resourcegroup="rg-np-version-upgrade-s4lpkb-nckb2tfdx9sw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-s4lpkb-nckb2tfdx9sw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-s4lpkb/nodePools/npupgrade-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x15ccc8e3e260>: 
    failed to start nodepool "npupgrade-4-21" update in cluster "np-version-upgrade-cluster-s4lpkb" resourcegroup="rg-np-version-upgrade-s4lpkb-nckb2tfdx9sw": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-s4lpkb-nckb2tfdx9sw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-s4lpkb/nodePools/npupgrade-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x2ec55b7f8360>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vqbwskk476qx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x2ec55b7f8360>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-vqbwskk476qx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-9h95dn" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0x1b712a5307b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-fb447pvfc5mf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-9h95dn
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-20-9h95dn" on minor version 4.20
Unexpected error:
    <*exported.ResponseError | 0x1b712a5307b0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-fb447pvfc5mf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-9h95dn
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-7pp5zf" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0x3221c9e003f0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-dhsqvc8s4dts/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-7pp5zf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-21-7pp5zf" on minor version 4.21
Unexpected error:
    <*exported.ResponseError | 0x3221c9e003f0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-dhsqvc8s4dts/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-7pp5zf
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x3f4d1e67e100>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-5nlftxqfx9mj": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-5nlftxqfx9mj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:189]: failed to update AZ nodepool autoscale-az max replicas to 4
Unexpected error:
    <*fmt.wrapError | 0x3f4d1e67e100>: 
    failed to start nodepool "autoscale-az" update in cluster "np-autoscale-cluster" resourcegroup="np-autoscaling-5nlftxqfx9mj": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-5nlftxqfx9mj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster/nodePools/autoscale-az
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.41 to 4.22.17
Unexpected error:
    <*fmt.wrapError | 0x22aa54096dc0>: 
    failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-qs6slk" resourcegroup="rg-np-skip-minor-qs6slk-svmgzjwq29hn": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-qs6slk-svmgzjwq29hn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-qs6slk/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:489]: failed to upgrade nodepool nps-420-422 from 4.20.41 to 4.22.17
Unexpected error:
    <*fmt.wrapError | 0x22aa54096dc0>: 
    failed to start nodepool "nps-420-422" update in cluster "np-skip-minor-qs6slk" resourcegroup="rg-np-skip-minor-qs6slk-svmgzjwq29hn": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-qs6slk-svmgzjwq29hn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-qs6slk/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x6f0cf085080>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-6zf46c" resourcegroup="rg-np-dg-minor-6zf46c-hll6hfr44s9d": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-6zf46c-hll6hfr44s9d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-6zf46c/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:768]: failed to update nodepool npdg-422-420 to downgrade target 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x6f0cf085080>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-6zf46c" resourcegroup="rg-np-dg-minor-6zf46c-hll6hfr44s9d": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-6zf46c-hll6hfr44s9d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-6zf46c/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:628]: failed to update nodepool npdg-4-21 to downgrade target 4.21.35
Unexpected error:
    <*fmt.wrapError | 0x1663af28c900>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-qlgw5r" resourcegroup="rg-np-downgrade-qlgw5r-fvqqvhgtclmz": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-qlgw5r-fvqqvhgtclmz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-qlgw5r/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:628]: failed to update nodepool npdg-4-21 to downgrade target 4.21.35
Unexpected error:
    <*fmt.wrapError | 0x1663af28c900>: 
    failed to start nodepool "npdg-4-21" update in cluster "np-downgrade-qlgw5r" resourcegroup="rg-np-downgrade-qlgw5r-fvqqvhgtclmz": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-qlgw5r-fvqqvhgtclmz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-qlgw5r/nodePools/npdg-4-21
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-j44hdg" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xf1601f277d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-7nd955rpcdtq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-j44hdg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-23-j44hdg" to 4.23
Unexpected error:
    <*exported.ResponseError | 0xf1601f277d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-7nd955rpcdtq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-j44hdg
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-s865cq" to 4.21
Unexpected error:
    <*exported.ResponseError | 0x29e047bb9ce0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-84fl6x5gkgjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-s865cq
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-21-s865cq" to 4.21
Unexpected error:
    <*exported.ResponseError | 0x29e047bb9ce0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-84fl6x5gkgjr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-s865cq
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x172168187dd0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-ffsspp7xdxbw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x172168187dd0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-ffsspp7xdxbw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0xad5396d32a0>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-wl7vjdkvvpf7": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-wl7vjdkvvpf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapError | 0xad5396d32a0>: 
    failed to start nodepool "np-1" update in cluster "cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-wl7vjdkvvpf7": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-wl7vjdkvvpf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints/nodePools/np-1
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-kqzff9" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xe01aaa2bbc0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7pmn5ng5wrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-kqzff9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-5-0-kqzff9" on minor version 5.0
Unexpected error:
    <*exported.ResponseError | 0xe01aaa2bbc0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-p7pmn5ng5wrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-kqzff9
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-g5xfmb" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x2d844c21d830>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-5xdvnklmwfq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-g5xfmb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-22-g5xfmb" on minor version 4.22
Unexpected error:
    <*exported.ResponseError | 0x2d844c21d830>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-5xdvnklmwfq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-g5xfmb
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x669cc1b95c0>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-mwkm86" resourcegroup="rg-np-version-upgrade-mwkm86-hh468p9mpt76": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-mwkm86-hh468p9mpt76/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-mwkm86/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.21.36
Unexpected error:
    <*fmt.wrapError | 0x669cc1b95c0>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-mwkm86" resourcegroup="rg-np-version-upgrade-mwkm86-hh468p9mpt76": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-mwkm86-hh468p9mpt76/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-mwkm86/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-qldg5l" to 5.0
Unexpected error:
    <*exported.ResponseError | 0x200557060510>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-hldpsj9c8292/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-qldg5l
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-0-qldg5l" to 5.0
Unexpected error:
    <*exported.ResponseError | 0x200557060510>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-hldpsj9c8292/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-qldg5l
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-pwmrlt" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x364694577140>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-bcxf7z6jvkdm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-pwmrlt
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-5-1-pwmrlt" to 5.1
Unexpected error:
    <*exported.ResponseError | 0x364694577140>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-bcxf7z6jvkdm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-pwmrlt
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-t4k2nv" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x9a9211d2ff0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-wdg9f8phhv6h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-t4k2nv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:157]: failed to enable immediate z-stream updates for cluster "cluster-zstream-4-23-t4k2nv" on minor version 4.23
Unexpected error:
    <*exported.ResponseError | 0x9a9211d2ff0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-23-wdg9f8phhv6h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-23-t4k2nv
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.z-stream-update-policy\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.z-stream-update-policy]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x1508ab4d7940>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-lc4jfj" resourcegroup="rg-np-version-upgrade-lc4jfj-qb5cmdpk8ppz": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lc4jfj-qb5cmdpk8ppz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-lc4jfj/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-20 to version 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x1508ab4d7940>: 
    failed to start nodepool "npupgrade-4-20" update in cluster "np-version-upgrade-cluster-lc4jfj" resourcegroup="rg-np-version-upgrade-lc4jfj-qb5cmdpk8ppz": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-lc4jfj-qb5cmdpk8ppz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-lc4jfj/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-9d5886" to 4.22
Unexpected error:
    <*exported.ResponseError | 0x2e46520ddbc0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-k6tc5l862fq5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-9d5886
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:167]: failed to trigger y-stream upgrade of cluster "cp-ystream-upgrade-4-22-9d5886" to 4.22
Unexpected error:
    <*exported.ResponseError | 0x2e46520ddbc0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-k6tc5l862fq5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-9d5886
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x2554f7d0b4d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-6xchsw5qtmxd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x2554f7d0b4d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-6xchsw5qtmxd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.cluster.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.cluster.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:349]: failed to upgrade nodepool npnoedge-4-20 from 4.20.40 to 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x18577971d620>: 
    failed to start nodepool "npnoedge-4-20" update in cluster "np-noedge-j47stk" resourcegroup="rg-np-noedge-j47stk-c7scxw66l4hg": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-j47stk-c7scxw66l4hg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-j47stk/nodePools/npnoedge-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:349]: failed to upgrade nodepool npnoedge-4-20 from 4.20.40 to 4.20.41
Unexpected error:
    <*fmt.wrapError | 0x18577971d620>: 
    failed to start nodepool "npnoedge-4-20" update in cluster "np-noedge-j47stk" resourcegroup="rg-np-noedge-j47stk-c7scxw66l4hg": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-j47stk-c7scxw66l4hg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-j47stk/nodePools/npnoedge-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"aro-hcp.experimental.nodepool.max-update-duration\": unrecognized experimental tag",
        "target": "tags[aro-hcp.experimental.nodepool.max-update-duration]"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x3fc6670d91a0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-mgmsfgq99bg9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0x3fc6670d91a0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-mgmsfgq99bg9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x3ecd88409b60>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-fjsmwtbqw48w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0x3ecd88409b60>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-fjsmwtbqw48w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x179ee7733d70>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-nxkg9dgmqghf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0x179ee7733d70>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-nxkg9dgmqghf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x3f9b94218c90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-jr65q449l4wp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0x3f9b94218c90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-jr65q449l4wp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x16ef84387050>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-8hwhxx82br9m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0x16ef84387050>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-8hwhxx82br9m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x13cc8f58b420>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-sz5gdnfzqhdd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c72bb6d-878b-457e-bf39-56f98851abda
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c72bb6d-878b-457e-bf39-56f98851abda",
      "name": "2c72bb6d-878b-457e-bf39-56f98851abda",
      "status": "Failed",
      "startTime": "2026-10-09T06:39:07.400427011Z",
      "endTime": "2026-10-09T07:08:09.774067703Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-sz5gdnfzqhdd--managed/providers/Microsoft.Compute/virtualMachines/y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-sz5gdnfzqhdd--managed/providers/Microsoft.Compute/virtualMachines/y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:117]: failed to create node pool cordon-np
Unexpected error:
    <*fmt.wrapError | 0x13cc8f58b420>: 
    failed to create NodePool cordon-np: failed waiting for nodepool="cordon-np" for cluster "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-sz5gdnfzqhdd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c72bb6d-878b-457e-bf39-56f98851abda
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeadlineExceeded
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/2c72bb6d-878b-457e-bf39-56f98851abda",
      "name": "2c72bb6d-878b-457e-bf39-56f98851abda",
      "status": "Failed",
      "startTime": "2026-10-09T06:39:07.400427011Z",
      "endTime": "2026-10-09T07:08:09.774067703Z",
      "error": {
        "code": "DeadlineExceeded",
        "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-sz5gdnfzqhdd--managed/providers/Microsoft.Compute/virtualMachines/y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-sz5gdnfzqhdd--managed/providers/Microsoft.Compute/virtualMachines/y1p1d9s3x8k4l0j-cordon-np-ht298-w2dcd to exist\n"
      }
    }
    --------------------------------------------------------------------------------
    
    <Azure error internals omitted>
occurred