No runs match your search.
Environment: DEV
| Time (UTC) | Runtime | Job | Region | Failed at | Result | PR | Failed tests | Details |
|---|---|---|---|---|---|---|---|---|
| 2h 25m 32s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | centralus | alert | failed | #7495 (open) | 2 | alert (2)alert [infra] Cosmos DB Normalized RU Consumption Burst High - <cosmos-account> fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-14T00:40:15Z Ended: 2026-10-14T01:10:12Z Severity: Sev3 Description: Cosmos DB normalized RU consumption is above 95% averaged over a 5-minute window, evaluated every minute. This catches short RU-saturation bursts (e.g. mass cluster creation at the start of E2E tests) that exceed provisioned throughput and can lead to 429 throttling before the 10-minute sustained alert fires. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units alert [infra] Cosmos DB Normalized RU Consumption High - <cosmos-account> fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-14T00:46:13Z Ended: 2026-10-14T01:18:14Z Severity: Sev3 Description: Cosmos DB normalized RU consumption is above 70% averaged over a 15-minute window, evaluated every minute. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units |
|
| 2h 17m 37s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | n/a | passed | #7469 (merged) | 0 | n/a | |
| 2h 3m 44s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | centralus | e2e | failed | #7451 (open) | 1 | e2e (1)clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url> Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:364]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x283328822c40>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:53294->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:53294->172.30.0.10:53: read: connection refused",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:364]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0x283328822c40>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:53294->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:53294->172.30.0.10:53: read: connection refused",
} |
|
| 2h 12m 55s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | centralus | e2e | failed | #6712 (open) | 1 | e2e (1)cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded" Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-wvbxxwdg6hgk/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-wvbxxwdg6hgk/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failed |
|
| 2h 15m 32s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | alert, e2e | failed | #7509 (merged) | 4 | e2e (1)ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-8fsm9cfscsm2/cluster-candidate-5-0-gcnmsr should provision
Unexpected error:
<*fmt.wrapError | 0x31dfd85b2460>:
failed to create HCP cluster cluster-candidate-5-0-gcnmsr: failed waiting for cluster="cluster-candidate-5-0-gcnmsr" in resourcegroup="rg-candidate-5-0-8fsm9cfscsm2" to finish creating: GET https://rp.j3394432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b3f80f07-6711-40dc-aa63-cd1bb0fc5de6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b3f80f07-6711-40dc-aa63-cd1bb0fc5de6",
"name": "b3f80f07-6711-40dc-aa63-cd1bb0fc5de6",
"status": "Failed",
"startTime": "2026-10-13T22:56:29.235009494Z",
"endTime": "2026-10-13T23:25:36.777262748Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3394432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b3f80f07-6711-40dc-aa63-cd1bb0fc5de6
correlation ID: fd6f57aa-c468-479d-a56c-8186a1e8023b
client request ID: 427c0be1-3302-4f9f-a711-14e8b93546be
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-8fsm9cfscsm2/cluster-candidate-5-0-gcnmsr should provision
Unexpected error:
<*fmt.wrapError | 0x31dfd85b2460>:
failed to create HCP cluster cluster-candidate-5-0-gcnmsr: failed waiting for cluster="cluster-candidate-5-0-gcnmsr" in resourcegroup="rg-candidate-5-0-8fsm9cfscsm2" to finish creating: GET https://rp.j3394432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b3f80f07-6711-40dc-aa63-cd1bb0fc5de6
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b3f80f07-6711-40dc-aa63-cd1bb0fc5de6",
"name": "b3f80f07-6711-40dc-aa63-cd1bb0fc5de6",
"status": "Failed",
"startTime": "2026-10-13T22:56:29.235009494Z",
"endTime": "2026-10-13T23:25:36.777262748Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3394432.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b3f80f07-6711-40dc-aa63-cd1bb0fc5de6
correlation ID: fd6f57aa-c468-479d-a56c-8186a1e8023b
client request ID: 427c0be1-3302-4f9f-a711-14e8b93546be
occurredalert (3)alert [hcp] KubeStatefulSetReplicasMismatch fired Show raw failurealert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-13T23:17:32Z Ended: 2026-10-13T23:44:30Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3394432-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3394432/providers/microsoft.monitor/accounts/hcps-j3394432", namespace="ocm-arohcpci01-2te6i7gkbepqvjtjr55req1psul8267o-l7d9p9t3f9q6b2m", pod="arohcp-monitor-kube-state-metrics-7988dc648c-l4xqr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2te6i7gkbepqvjtjr55req1psul8267o-l7d9p9t3f9q6b2m/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-13T23:17:32Z Ended: 2026-10-13T23:43:31Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3394432-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3394432/providers/microsoft.monitor/accounts/hcps-j3394432", namespace="ocm-arohcpci01-2te6i7gkbepqvjtjr55req1psul8267o-l7d9p9t3f9q6b2m", pod="arohcp-monitor-kube-state-metrics-7988dc648c-l4xqr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2te6i7gkbepqvjtjr55req1psul8267o-l7d9p9t3f9q6b2m/etcd has not matched the expected number of replicas for longer than 15 minutes. alert [hcp] KubeDeploymentReplicasMismatch fired Show raw failurealert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-13T23:32:29Z Ended: 2026-10-13T23:45:31Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3394432-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3394432/providers/microsoft.monitor/accounts/hcps-j3394432", namespace="ocm-arohcpci01-2te6i7gkbepqvjtjr55req1psul8267o-l7d9p9t3f9q6b2m", pod="arohcp-monitor-kube-state-metrics-7988dc648c-l4xqr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2te6i7gkbepqvjtjr55req1psul8267o-l7d9p9t3f9q6b2m/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-13T23:32:29Z Ended: 2026-10-13T23:45:31Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3394432-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3394432/providers/microsoft.monitor/accounts/hcps-j3394432", namespace="ocm-arohcpci01-2te6i7gkbepqvjtjr55req1psul8267o-l7d9p9t3f9q6b2m", pod="arohcp-monitor-kube-state-metrics-7988dc648c-l4xqr", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2te6i7gkbepqvjtjr55req1psul8267o-l7d9p9t3f9q6b2m/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert [hcp] KubePodCrashLooping fired Show raw failurealert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-13T23:20:28Z Ended: 2026-10-13T23:48:28Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3394432-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3394432/providers/microsoft.monitor/accounts/hcps-j3394432", namespace="ocm-arohcpci01-2te6i7gkbepqvjtjr55req1psul8267o-l7d9p9t3f9q6b2m", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="77588538-1895-449d-bb34-5ee7b9aabbd3" Description: Pod ocm-arohcpci01-2te6i7gkbepqvjtjr55req1psul8267o-l7d9p9t3f9q6b2m/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-13T23:20:28Z Ended: 2026-10-13T23:48:28Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3394432-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3394432/providers/microsoft.monitor/accounts/hcps-j3394432", namespace="ocm-arohcpci01-2te6i7gkbepqvjtjr55req1psul8267o-l7d9p9t3f9q6b2m", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="77588538-1895-449d-bb34-5ee7b9aabbd3" Description: Pod ocm-arohcpci01-2te6i7gkbepqvjtjr55req1psul8267o-l7d9p9t3f9q6b2m/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff"). |
|
| 2h 10m 2s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
centralus | n/a | passed | #7401 (merged) | 0 | n/a | |
| 2h 14m 15s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | westus3 | e2e | failed | #7477 (closed) | 3 | e2e (3)IngressController loadBalancer config is nil Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:198]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
<*errors.errorString | 0x374a5e1794b0>:
IngressController loadBalancer config is nil
{
s: "IngressController loadBalancer config is nil",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:198]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
<*errors.errorString | 0x374a5e1794b0>:
IngressController loadBalancer config is nil
{
s: "IngressController loadBalancer config is nil",
}IngressController loadBalancer config is nil Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:198]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
<*errors.errorString | 0x284568f46bc0>:
IngressController loadBalancer config is nil
{
s: "IngressController loadBalancer config is nil",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:198]: default ingress certificate scenario requires public ingress
Expected success, but got an error:
<*errors.errorString | 0x284568f46bc0>:
IngressController loadBalancer config is nil
{
s: "IngressController loadBalancer config is nil",
}| take 101, Post "<url>": context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5d7srqg64hdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2b033ad20c80>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5d7srqg64hdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5d7srqg64hdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-13T22:55:07.0962439Z);
let desires = cosmosResourceSnapshots
| where resourceID startswith strcat(target, "/")
| where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
| where isempty(content.deletionTimestamp)
| extend manifest = content.properties.status.kubeContent
| where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
managementCluster = tolower(tostring(content.properties.spec.managementCluster))
| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
let fleet = cosmosResourceSnapshots
| where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
| where isempty(content.deletionTimestamp)
| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
| where isnotempty(clusterName)
| project environment, region, sourceCluster, managementCluster, cluster = clusterName;
let hcp = desires
| join kind=inner fleet on environment, region, sourceCluster, managementCluster
| distinct environment, region, cluster, namespace;
let inventory = materialize(kubernetesResourceSnapshots
| where timestamp >= start and objectKind in ("Pod", "Deployment")
| join kind=leftsemi hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
| where event != "Delete" and isempty(object.metadata.deletionTimestamp));
let pods = inventory
| where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
let deployments = inventory
| where objectKind == "Deployment" and name == "router"
| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
let summaries = containerLogs
| where timestamp >= start and container_name == "swift-recorder"
| where log.controller_name == "swift-router-check" and log.field == "summary"
| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
| join kind=leftsemi hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
hcp
| join kind=leftouter deployments on environment, region, cluster, namespace
| join kind=leftouter counts on environment, region, cluster, namespace
| join kind=leftouter pods on environment, region, cluster, namespace
| join kind=leftouter summaries on environment, region, cluster, namespace, UID
| project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
| order by Scope asc, UID asc
| take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5d7srqg64hdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5d7srqg64hdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-13T22:55:07.0962439Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*fmt.wrapError | 0x2b033ad20c60>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5d7srqg64hdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-13T22:55:07.0962439Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2b0339c5a8c0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2b033ad20c40>{
msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5d7srqg64hdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-13T22:55:07.0962439Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2b033bf13d10>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5d7srqg64hdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x2b033ad20c80>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5d7srqg64hdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5d7srqg64hdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-13T22:55:07.0962439Z);
let desires = cosmosResourceSnapshots
| where resourceID startswith strcat(target, "/")
| where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
| where isempty(content.deletionTimestamp)
| extend manifest = content.properties.status.kubeContent
| where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
managementCluster = tolower(tostring(content.properties.spec.managementCluster))
| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
let fleet = cosmosResourceSnapshots
| where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
| where isempty(content.deletionTimestamp)
| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
| where isnotempty(clusterName)
| project environment, region, sourceCluster, managementCluster, cluster = clusterName;
let hcp = desires
| join kind=inner fleet on environment, region, sourceCluster, managementCluster
| distinct environment, region, cluster, namespace;
let inventory = materialize(kubernetesResourceSnapshots
| where timestamp >= start and objectKind in ("Pod", "Deployment")
| join kind=leftsemi hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
| where event != "Delete" and isempty(object.metadata.deletionTimestamp));
let pods = inventory
| where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
let deployments = inventory
| where objectKind == "Deployment" and name == "router"
| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
let summaries = containerLogs
| where timestamp >= start and container_name == "swift-recorder"
| where log.controller_name == "swift-router-check" and log.field == "summary"
| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
| join kind=leftsemi hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
hcp
| join kind=leftouter deployments on environment, region, cluster, namespace
| join kind=leftouter counts on environment, region, cluster, namespace
| join kind=leftouter pods on environment, region, cluster, namespace
| join kind=leftouter summaries on environment, region, cluster, namespace, UID
| project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
| order by Scope asc, UID asc
| take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5d7srqg64hdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.001s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5d7srqg64hdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-13T22:55:07.0962439Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*fmt.wrapError | 0x2b033ad20c60>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5d7srqg64hdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-13T22:55:07.0962439Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x2b0339c5a8c0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x2b033ad20c40>{
msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-5d7srqg64hdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-13T22:55:07.0962439Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x2b033bf13d10>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
}
occurred |
|
| 2h 18m 1s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | e2e | failed | #7488 (merged) | 1 | e2e (1)failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-12T23:09:06 Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 604.992s.
Expected success, but got an error:
<*fmt.wrapError | 0x39f5f91e0c80>:
failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-12T23:09:06.1094684Z);
let _endTime = datetime(2026-10-13T23:09:06.1094687Z);
// Extract cluster ID from CS logs using cid field
let _cs_cluster_id = toscalar(
database('ServiceLogs').table('clustersServiceLogs')
| where timestamp between (_startTime .. _endTime)
| where resource_id has 'kusto-logs-nvqd2j9ncsfl'
| where isnotempty(log.cid)
| distinct tostring(log.cid)
| limit 1
);
// Find all Maestro resource IDs for this cluster
let maestro_resource_ids = (
database('ServiceLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has "maestro"
| extend logStr = tostring(log)
| where logStr has _cs_cluster_id
| extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
| where isnotempty(resourceId)
| distinct resourceId
);
// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
union
(
// Maestro logs: Resource bundle operations for this cluster and its associated resources
// Includes: resource creation, updates, deletions, status events, gRPC communications
database('ServiceLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has "maestro"
| extend logStr = tostring(log)
| where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
| project timestamp, container_name, msg = logStr, log
),
(
// Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
// Includes the CR reconciliation logs
database('ServiceLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has "hypershift"
| extend logStr = tostring(log)
| where logStr has _cs_cluster_id
| project timestamp, container_name, msg = logStr, log
),
(
// ACM Agent logs: ManifestWork reconciliation on the managed cluster
// Includes: work agent operations, manifest application, status feedback
database('ServiceLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has "open-cluster-management-agent"
| extend logStr = tostring(log)
| where logStr has _cs_cluster_id
| project timestamp, container_name, msg = logStr, log
)
| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-13T23:09:29.6842855Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea", ActivityId:"0d59e32d-82d3-46a4-953e-e100fb2ccef8", SubActivityId:"0d59e32d-82d3-46a4-953e-e100fb2ccef8", ActivityType:"GW.Http.CallContext", ParentActivityId:"0d59e32d-82d3-46a4-953e-e100fb2ccef8", ActivityStack:"(Activity stack: CRID=KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea ARID=0d59e32d-82d3-46a4-953e-e100fb2ccef8 > GW.Http.CallContext/0d59e32d-82d3-46a4-953e-e100fb2ccef8)"}, IsPermanent:false})
{
msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T23:09:06.1094684Z);\nlet _endTime = datetime(2026-10-13T23:09:06.1094687Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-nvqd2j9ncsfl'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T23:09:29.6842855Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea\", ActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", SubActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityStack:\"(Activity stack: CRID=KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea ARID=0d59e32d-82d3-46a4-953e-e100fb2ccef8 > GW.Http.CallContext/0d59e32d-82d3-46a4-953e-e100fb2ccef8)\"}, IsPermanent:false})",
err: <*errors.joinError | 0x39f5f8768288>{
errs: [
<*fmt.wrapError | 0x39f5f91e0c60>{
msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T23:09:06.1094684Z);\nlet _endTime = datetime(2026-10-13T23:09:06.1094687Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-nvqd2j9ncsfl'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T23:09:29.6842855Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea\", ActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", SubActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityStack:\"(Activity stack: CRID=KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea ARID=0d59e32d-82d3-46a4-953e-e100fb2ccef8 > GW.Http.CallContext/0d59e32d-82d3-46a4-953e-e100fb2ccef8)\"}, IsPermanent:false})",
err: <*fmt.wrapError | 0x39f5f91e0c40>{
msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T23:09:06.1094684Z);\nlet _endTime = datetime(2026-10-13T23:09:06.1094687Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-nvqd2j9ncsfl'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T23:09:29.6842855Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea\", ActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", SubActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityStack:\"(Activity stack: CRID=KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea ARID=0d59e32d-82d3-46a4-953e-e100fb2ccef8 > GW.Http.CallContext/0d59e32d-82d3-46a4-953e-e100fb2ccef8)\"}, IsPermanent:false})",
err: <*fmt.wrapError | 0x39f5f91e0ba0>{
msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T23:09:06.1094684Z);\nlet _endTime = datetime(2026-10-13T23:09:06.1094687Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-nvqd2j9ncsfl'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T23:09:29.6842855Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea\", ActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", SubActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityStack:\"(Activity stack: CRID=KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea ARID=0d59e32d-82d3-46a4-953e-e100fb2ccef8 > GW.Http.CallContext/0d59e32d-82d3-46a4-953e-e100fb2ccef8)\"}, IsPermanent:false})",
err: <*fmt.wrapError | 0x39f5f91e0b60>{
msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T23:09:06.1094684Z);\nlet _endTime = datetime(2026-10-13T23:09:06.1094687Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-nvqd2j9ncsfl'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T23:09:29.6842855Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea\", ActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", SubActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityStack:\"(Activity stack: CRID=KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea ARID=0d59e32d-82d3-46a4-953e-e100fb2ccef8 > GW.Http.CallContext/0d59e32d-82d3-46a4-953e-e100fb2ccef8)\"}, IsPermanent:false})",
err: <*v2.OneApiError | 0x39f5f6b7ee00>{
ErrorMessage: {
Code: "LimitsExceeded",
Message: "Request is invalid and cannot be executed.",
Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
Context: {
Timestamp: ...,
ServiceAlias: ...,
MachineName: ...,
ProcessName: ...,
ProcessId: ...,
ThreadId: ...,
ClientRequestId: ...,
ActivityId: ...,
SubActivityId: ...,
ActivityType: ...,
ParentActivityId: ...,
ActivityStack: ...,
},
IsPermanent: false,
},
},
},
},
},
},
],
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 604.992s.
Expected success, but got an error:
<*fmt.wrapError | 0x39f5f91e0c80>:
failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-12T23:09:06.1094684Z);
let _endTime = datetime(2026-10-13T23:09:06.1094687Z);
// Extract cluster ID from CS logs using cid field
let _cs_cluster_id = toscalar(
database('ServiceLogs').table('clustersServiceLogs')
| where timestamp between (_startTime .. _endTime)
| where resource_id has 'kusto-logs-nvqd2j9ncsfl'
| where isnotempty(log.cid)
| distinct tostring(log.cid)
| limit 1
);
// Find all Maestro resource IDs for this cluster
let maestro_resource_ids = (
database('ServiceLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has "maestro"
| extend logStr = tostring(log)
| where logStr has _cs_cluster_id
| extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
| where isnotempty(resourceId)
| distinct resourceId
);
// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
union
(
// Maestro logs: Resource bundle operations for this cluster and its associated resources
// Includes: resource creation, updates, deletions, status events, gRPC communications
database('ServiceLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has "maestro"
| extend logStr = tostring(log)
| where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
| project timestamp, container_name, msg = logStr, log
),
(
// Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
// Includes the CR reconciliation logs
database('ServiceLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has "hypershift"
| extend logStr = tostring(log)
| where logStr has _cs_cluster_id
| project timestamp, container_name, msg = logStr, log
),
(
// ACM Agent logs: ManifestWork reconciliation on the managed cluster
// Includes: work agent operations, manifest application, status feedback
database('ServiceLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has "open-cluster-management-agent"
| extend logStr = tostring(log)
| where logStr has _cs_cluster_id
| project timestamp, container_name, msg = logStr, log
)
| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-13T23:09:29.6842855Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea", ActivityId:"0d59e32d-82d3-46a4-953e-e100fb2ccef8", SubActivityId:"0d59e32d-82d3-46a4-953e-e100fb2ccef8", ActivityType:"GW.Http.CallContext", ParentActivityId:"0d59e32d-82d3-46a4-953e-e100fb2ccef8", ActivityStack:"(Activity stack: CRID=KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea ARID=0d59e32d-82d3-46a4-953e-e100fb2ccef8 > GW.Http.CallContext/0d59e32d-82d3-46a4-953e-e100fb2ccef8)"}, IsPermanent:false})
{
msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T23:09:06.1094684Z);\nlet _endTime = datetime(2026-10-13T23:09:06.1094687Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-nvqd2j9ncsfl'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T23:09:29.6842855Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea\", ActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", SubActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityStack:\"(Activity stack: CRID=KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea ARID=0d59e32d-82d3-46a4-953e-e100fb2ccef8 > GW.Http.CallContext/0d59e32d-82d3-46a4-953e-e100fb2ccef8)\"}, IsPermanent:false})",
err: <*errors.joinError | 0x39f5f8768288>{
errs: [
<*fmt.wrapError | 0x39f5f91e0c60>{
msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T23:09:06.1094684Z);\nlet _endTime = datetime(2026-10-13T23:09:06.1094687Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-nvqd2j9ncsfl'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T23:09:29.6842855Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea\", ActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", SubActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityStack:\"(Activity stack: CRID=KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea ARID=0d59e32d-82d3-46a4-953e-e100fb2ccef8 > GW.Http.CallContext/0d59e32d-82d3-46a4-953e-e100fb2ccef8)\"}, IsPermanent:false})",
err: <*fmt.wrapError | 0x39f5f91e0c40>{
msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T23:09:06.1094684Z);\nlet _endTime = datetime(2026-10-13T23:09:06.1094687Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-nvqd2j9ncsfl'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T23:09:29.6842855Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea\", ActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", SubActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityStack:\"(Activity stack: CRID=KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea ARID=0d59e32d-82d3-46a4-953e-e100fb2ccef8 > GW.Http.CallContext/0d59e32d-82d3-46a4-953e-e100fb2ccef8)\"}, IsPermanent:false})",
err: <*fmt.wrapError | 0x39f5f91e0ba0>{
msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T23:09:06.1094684Z);\nlet _endTime = datetime(2026-10-13T23:09:06.1094687Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-nvqd2j9ncsfl'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T23:09:29.6842855Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea\", ActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", SubActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityStack:\"(Activity stack: CRID=KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea ARID=0d59e32d-82d3-46a4-953e-e100fb2ccef8 > GW.Http.CallContext/0d59e32d-82d3-46a4-953e-e100fb2ccef8)\"}, IsPermanent:false})",
err: <*fmt.wrapError | 0x39f5f91e0b60>{
msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T23:09:06.1094684Z);\nlet _endTime = datetime(2026-10-13T23:09:06.1094687Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-nvqd2j9ncsfl'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T23:09:29.6842855Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea\", ActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", SubActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"0d59e32d-82d3-46a4-953e-e100fb2ccef8\", ActivityStack:\"(Activity stack: CRID=KGC.execute;6eceef39-9526-457e-97d9-feedc18102ea ARID=0d59e32d-82d3-46a4-953e-e100fb2ccef8 > GW.Http.CallContext/0d59e32d-82d3-46a4-953e-e100fb2ccef8)\"}, IsPermanent:false})",
err: <*v2.OneApiError | 0x39f5f6b7ee00>{
ErrorMessage: {
Code: "LimitsExceeded",
Message: "Request is invalid and cannot be executed.",
Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
Context: {
Timestamp: ...,
ServiceAlias: ...,
MachineName: ...,
ProcessName: ...,
ProcessId: ...,
ThreadId: ...,
ClientRequestId: ...,
ActivityId: ...,
SubActivityId: ...,
ActivityType: ...,
ParentActivityId: ...,
ActivityStack: ...,
},
IsPermanent: false,
},
},
},
},
},
},
],
},
} |
|
| 2h 14m 46s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | e2e | failed | #7248 (merged) | 2 | e2e (2)failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x2b2721dce700>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-xq96765qgwxz" to finish creating: GET https://rp.j9424512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70cccd80-322a-41c6-aef6-dff76071386f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70cccd80-322a-41c6-aef6-dff76071386f",
"name": "70cccd80-322a-41c6-aef6-dff76071386f",
"status": "Failed",
"startTime": "2026-10-13T22:47:02.133519323Z",
"endTime": "2026-10-13T23:16:06.91066671Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x8s4a0b7o3h5z7m-np-delete-main-8qs7b-hs6qs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-xq96765qgwxz--managed/providers/Microsoft.Compute/virtualMachines/x8s4a0b7o3h5z7m-np-delete-main-8qs7b-hs6qs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine x8s4a0b7o3h5z7m-np-delete-main-8qs7b-hs6qs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-xq96765qgwxz--managed/providers/Microsoft.Compute/virtualMachines/x8s4a0b7o3h5z7m-np-delete-main-8qs7b-hs6qs to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9424512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70cccd80-322a-41c6-aef6-dff76071386f
correlation ID: 10fb6c26-b492-4ad7-bd1d-956c35d26f43
client request ID: 62230a28-0708-4526-b5b0-b1e90cbbcfa0
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x2b2721dce700>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-xq96765qgwxz" to finish creating: GET https://rp.j9424512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70cccd80-322a-41c6-aef6-dff76071386f
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70cccd80-322a-41c6-aef6-dff76071386f",
"name": "70cccd80-322a-41c6-aef6-dff76071386f",
"status": "Failed",
"startTime": "2026-10-13T22:47:02.133519323Z",
"endTime": "2026-10-13T23:16:06.91066671Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine x8s4a0b7o3h5z7m-np-delete-main-8qs7b-hs6qs: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-xq96765qgwxz--managed/providers/Microsoft.Compute/virtualMachines/x8s4a0b7o3h5z7m-np-delete-main-8qs7b-hs6qs to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine x8s4a0b7o3h5z7m-np-delete-main-8qs7b-hs6qs: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-xq96765qgwxz--managed/providers/Microsoft.Compute/virtualMachines/x8s4a0b7o3h5z7m-np-delete-main-8qs7b-hs6qs to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9424512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/70cccd80-322a-41c6-aef6-dff76071386f
correlation ID: 10fb6c26-b492-4ad7-bd1d-956c35d26f43
client request ID: 62230a28-0708-4526-b5b0-b1e90cbbcfa0cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded" Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-qrdm5stt6s2r/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-qrdm5stt6s2r/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failed |
|
| 2h 15m 25s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | centralus | n/a | passed | #7278 (open) | 0 | n/a | |
| 2h 9m 2s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | centralus | alert, e2e | failed | #7495 (open) | 4 | e2e (1)failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x1a46c002a160>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-bvw879vk9kf2" to finish creating: GET https://rp.j2816128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/96d5a19d-7a94-4c90-b6f3-78fa73f80a64
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/96d5a19d-7a94-4c90-b6f3-78fa73f80a64",
"name": "96d5a19d-7a94-4c90-b6f3-78fa73f80a64",
"status": "Failed",
"startTime": "2026-10-13T22:07:49.726995766Z",
"endTime": "2026-10-13T22:36:58.709896645Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w4b8j4s5n4t4f5w-np-delete-main-k5rps-jn5mr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-bvw879vk9kf2--managed/providers/Microsoft.Compute/virtualMachines/w4b8j4s5n4t4f5w-np-delete-main-k5rps-jn5mr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w4b8j4s5n4t4f5w-np-delete-main-k5rps-jn5mr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-bvw879vk9kf2--managed/providers/Microsoft.Compute/virtualMachines/w4b8j4s5n4t4f5w-np-delete-main-k5rps-jn5mr to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2816128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/96d5a19d-7a94-4c90-b6f3-78fa73f80a64
correlation ID: 79ec83c3-f7b9-41f5-baf4-9c117904430d
client request ID: fa7965b1-cdd5-4c6b-bbfc-bfe81d721d6f
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x1a46c002a160>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-bvw879vk9kf2" to finish creating: GET https://rp.j2816128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/96d5a19d-7a94-4c90-b6f3-78fa73f80a64
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/96d5a19d-7a94-4c90-b6f3-78fa73f80a64",
"name": "96d5a19d-7a94-4c90-b6f3-78fa73f80a64",
"status": "Failed",
"startTime": "2026-10-13T22:07:49.726995766Z",
"endTime": "2026-10-13T22:36:58.709896645Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w4b8j4s5n4t4f5w-np-delete-main-k5rps-jn5mr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-bvw879vk9kf2--managed/providers/Microsoft.Compute/virtualMachines/w4b8j4s5n4t4f5w-np-delete-main-k5rps-jn5mr to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w4b8j4s5n4t4f5w-np-delete-main-k5rps-jn5mr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-bvw879vk9kf2--managed/providers/Microsoft.Compute/virtualMachines/w4b8j4s5n4t4f5w-np-delete-main-k5rps-jn5mr to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2816128.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/96d5a19d-7a94-4c90-b6f3-78fa73f80a64
correlation ID: 79ec83c3-f7b9-41f5-baf4-9c117904430d
client request ID: fa7965b1-cdd5-4c6b-bbfc-bfe81d721d6falert (3)alert [infra] Cosmos DB Normalized RU Consumption High - <cosmos-account> fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T22:09:34Z Ended: 2026-10-13T22:42:14Z Severity: Sev3 Description: Cosmos DB normalized RU consumption is above 70% averaged over a 15-minute window, evaluated every minute. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units alert [hcp] KubePodNotReady fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T22:19:01Z Ended: 2026-10-13T22:28:55Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2816128-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2te5m776ut8ij1hl0ecm8bji2rolvh54-y4r5q8u1c0g8x2m", pod="router-5654cb9d4d-gd8l4", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2te5m776ut8ij1hl0ecm8bji2rolvh54-y4r5q8u1c0g8x2m/router-5654cb9d4d-gd8l4 has been in a non-ready state for longer than 5 minutes. alert [infra] Cosmos DB Normalized RU Consumption Burst High - <cosmos-account> fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T22:03:21Z Ended: 2026-10-13T22:29:18Z Severity: Sev3 Description: Cosmos DB normalized RU consumption is above 95% averaged over a 5-minute window, evaluated every minute. This catches short RU-saturation bursts (e.g. mass cluster creation at the start of E2E tests) that exceed provisioned throughput and can lead to 429 throttling before the 10-minute sustained alert fires. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units |
|
| 2h 16m 25s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | e2e | failed | #7469 (merged) | 2 | e2e (2)| take 101, Post "<url>": context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qpjkt6pf45sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x389cde3b8e20>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qpjkt6pf45sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.007s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qpjkt6pf45sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-13T21:46:20.5229327Z);
let desires = cosmosResourceSnapshots
| where resourceID startswith strcat(target, "/")
| where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
| where isempty(content.deletionTimestamp)
| extend manifest = content.properties.status.kubeContent
| where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
managementCluster = tolower(tostring(content.properties.spec.managementCluster))
| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
let fleet = cosmosResourceSnapshots
| where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
| where isempty(content.deletionTimestamp)
| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
| where isnotempty(clusterName)
| project environment, region, sourceCluster, managementCluster, cluster = clusterName;
let hcp = desires
| join kind=inner fleet on environment, region, sourceCluster, managementCluster
| distinct environment, region, cluster, namespace;
let inventory = materialize(kubernetesResourceSnapshots
| where timestamp >= start and objectKind in ("Pod", "Deployment")
| join kind=leftsemi hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
| where event != "Delete" and isempty(object.metadata.deletionTimestamp));
let pods = inventory
| where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
let deployments = inventory
| where objectKind == "Deployment" and name == "router"
| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
let summaries = containerLogs
| where timestamp >= start and container_name == "swift-recorder"
| where log.controller_name == "swift-router-check" and log.field == "summary"
| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
| join kind=leftsemi hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
hcp
| join kind=leftouter deployments on environment, region, cluster, namespace
| join kind=leftouter counts on environment, region, cluster, namespace
| join kind=leftouter pods on environment, region, cluster, namespace
| join kind=leftouter summaries on environment, region, cluster, namespace, UID
| project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
| order by Scope asc, UID asc
| take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qpjkt6pf45sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.007s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qpjkt6pf45sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-13T21:46:20.5229327Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*fmt.wrapError | 0x389cde3b8de0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qpjkt6pf45sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-13T21:46:20.5229327Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x389cdd5785a0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x389cde3b8d60>{
msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qpjkt6pf45sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-13T21:46:20.5229327Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x389cdf50a8a0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:112]: expected complete healthy SWIFT router, ignition, KAS, DNS and worker checks for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qpjkt6pf45sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster
Unexpected error:
<*fmt.wrapError | 0x389cde3b8e20>:
VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qpjkt6pf45sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.007s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qpjkt6pf45sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster"; let start = datetime(2026-10-13T21:46:20.5229327Z);
let desires = cosmosResourceSnapshots
| where resourceID startswith strcat(target, "/")
| where resourceType =~ "microsoft.redhatopenshift/hcpopenshiftclusters/readdesires"
| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID
| where isempty(content.deletionTimestamp)
| extend manifest = content.properties.status.kubeContent
| where manifest.kind == "HostedCluster" and isempty(manifest.metadata.deletionTimestamp)
| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),
managementCluster = tolower(tostring(content.properties.spec.managementCluster))
| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)
| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, "-", hostedClusterName);
let fleet = cosmosResourceSnapshots
| where resourceType =~ "microsoft.redhatopenshift/stamps/managementclusters"
| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster
| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster
| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts
| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster
| where isempty(content.deletionTimestamp)
| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), "/")[-1])
| where isnotempty(clusterName)
| project environment, region, sourceCluster, managementCluster, cluster = clusterName;
let hcp = desires
| join kind=inner fleet on environment, region, sourceCluster, managementCluster
| distinct environment, region, cluster, namespace;
let inventory = materialize(kubernetesResourceSnapshots
| where timestamp >= start and objectKind in ("Pod", "Deployment")
| join kind=leftsemi hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name
| where event != "Delete" and isempty(object.metadata.deletionTimestamp));
let pods = inventory
| where objectKind == "Pod" and object.metadata.labels.app == "private-router" and object.status.phase == "Running"
| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);
let counts = pods | summarize Current = count() by environment, region, cluster, namespace;
let deployments = inventory
| where objectKind == "Deployment" and name == "router"
| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);
let summaries = containerLogs
| where timestamp >= start and container_name == "swift-recorder"
| where log.controller_name == "swift-router-check" and log.field == "summary"
| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)
| join kind=leftsemi hcp on environment, region, cluster, namespace
| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID
| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),
SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);
hcp
| join kind=leftouter deployments on environment, region, cluster, namespace
| join kind=leftouter counts on environment, region, cluster, namespace
| join kind=leftouter pods on environment, region, cluster, namespace
| join kind=leftouter summaries on environment, region, cluster, namespace, UID
| project Scope = strcat(environment, "/", region, "/", cluster, "/", namespace),
Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary
| order by Scope asc, UID asc
| take 101, Post "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query": context deadline exceeded
{
msg: "VerifySwiftRouterChecks /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qpjkt6pf45sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster timed out after 12m0.007s: query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qpjkt6pf45sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-13T21:46:20.5229327Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*fmt.wrapError | 0x389cde3b8de0>{
msg: "query SWIFT checks in https://hcp-dev-us-2.eastus2.kusto.windows.net/ServiceLogs: Op(OpQuery): Kind(KHTTPError): With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qpjkt6pf45sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-13T21:46:20.5229327Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*errors.Error | 0x389cdd5785a0>{
Op: 1,
Kind: 7,
Err: <*fmt.wrapError | 0x389cde3b8d60>{
msg: "With query: let target = \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/kusto-logs-qpjkt6pf45sb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/engineering-hcp-cluster\"; let start = datetime(2026-10-13T21:46:20.5229327Z);\nlet desires = cosmosResourceSnapshots\n| where resourceID startswith strcat(target, \"/\")\n| where resourceType =~ \"microsoft.redhatopenshift/hcpopenshiftclusters/readdesires\"\n| extend resourceID = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, resourceID, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, resourceID\n| where isempty(content.deletionTimestamp)\n| extend manifest = content.properties.status.kubeContent\n| where manifest.kind == \"HostedCluster\" and isempty(manifest.metadata.deletionTimestamp)\n| extend hostedClusterNamespace = tostring(manifest.metadata.namespace), hostedClusterName = tostring(manifest.metadata.name),\n managementCluster = tolower(tostring(content.properties.spec.managementCluster))\n| where isnotempty(hostedClusterNamespace) and isnotempty(hostedClusterName) and isnotempty(managementCluster)\n| project environment, region, sourceCluster, managementCluster, namespace = strcat(hostedClusterNamespace, \"-\", hostedClusterName);\nlet fleet = cosmosResourceSnapshots\n| where resourceType =~ \"microsoft.redhatopenshift/stamps/managementclusters\"\n| extend managementCluster = tolower(resourceID), ts = tolong(content._ts), sourceCluster = cluster\n| join kind=leftsemi desires on environment, region, sourceCluster, managementCluster\n| summarize arg_max(timestamp, *) by environment, region, sourceCluster, managementCluster, ts\n| summarize arg_max(ts, *) by environment, region, sourceCluster, managementCluster\n| where isempty(content.deletionTimestamp)\n| extend clusterName = tostring(split(tostring(content.properties.status.aksResourceID), \"/\")[-1])\n| where isnotempty(clusterName)\n| project environment, region, sourceCluster, managementCluster, cluster = clusterName;\nlet hcp = desires\n| join kind=inner fleet on environment, region, sourceCluster, managementCluster\n| distinct environment, region, cluster, namespace;\nlet inventory = materialize(kubernetesResourceSnapshots\n| where timestamp >= start and objectKind in (\"Pod\", \"Deployment\")\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, objectKind, name\n| where event != \"Delete\" and isempty(object.metadata.deletionTimestamp));\nlet pods = inventory\n| where objectKind == \"Pod\" and object.metadata.labels.app == \"private-router\" and object.status.phase == \"Running\"\n| project environment, region, cluster, namespace, UID = uid, Node = tostring(object.spec.nodeName);\nlet counts = pods | summarize Current = count() by environment, region, cluster, namespace;\nlet deployments = inventory\n| where objectKind == \"Deployment\" and name == \"router\"\n| project environment, region, cluster, namespace, Desired = tolong(object.spec.replicas);\nlet summaries = containerLogs\n| where timestamp >= start and container_name == \"swift-recorder\"\n| where log.controller_name == \"swift-router-check\" and log.field == \"summary\"\n| extend namespace = tostring(log.pod_namespace), UID = tostring(log.pod_uid)\n| join kind=leftsemi hcp on environment, region, cluster, namespace\n| summarize arg_max(timestamp, *) by environment, region, cluster, namespace, UID\n| project environment, region, cluster, namespace, UID, SummaryUID = tostring(log.pod_uid),\n SummaryNode = tostring(log.node_name), Pass = tostring(log.pass_id), At = timestamp, Summary = tostring(log.record);\nhcp\n| join kind=leftouter deployments on environment, region, cluster, namespace\n| join kind=leftouter counts on environment, region, cluster, namespace\n| join kind=leftouter pods on environment, region, cluster, namespace\n| join kind=leftouter summaries on environment, region, cluster, namespace, UID\n| project Scope = strcat(environment, \"/\", region, \"/\", cluster, \"/\", namespace),\n Desired = coalesce(Desired, 0), Current = coalesce(Current, 0), UID, Node, SummaryUID, SummaryNode, Pass, At, Summary\n| order by Scope asc, UID asc\n| take 101, Post \"https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query\": context deadline exceeded",
err: <*url.Error | 0x389cdf50a8a0>{
Op: "Post",
URL: "https://hcp-dev-us-2.eastus2.kusto.windows.net/v2/rest/query",
Err: <context.deadlineExceededError>{},
},
},
restErrMsg: nil,
decoded: nil,
permanent: false,
inner: nil,
},
},
}
occurredfailed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:139]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x2515216a5b20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-dpn7kns9h5s6" to finish creating: GET https://rp.j9256960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02ba10f3-0e8a-4716-94ab-581f07c35309
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02ba10f3-0e8a-4716-94ab-581f07c35309",
"name": "02ba10f3-0e8a-4716-94ab-581f07c35309",
"status": "Failed",
"startTime": "2026-10-13T21:54:35.192495222Z",
"endTime": "2026-10-13T22:23:37.481089236Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-fc8dz-zl2sj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-dpn7kns9h5s6--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-fc8dz-zl2sj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-fc8dz-zl2sj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-dpn7kns9h5s6--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-fc8dz-zl2sj to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9256960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02ba10f3-0e8a-4716-94ab-581f07c35309
correlation ID: 69c597ca-4a5d-45f4-be8b-542aa16b2e4b
client request ID: 15111503-9313-4b39-bc7c-c0bfc7ad586b
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:139]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x2515216a5b20>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-dpn7kns9h5s6" to finish creating: GET https://rp.j9256960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02ba10f3-0e8a-4716-94ab-581f07c35309
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02ba10f3-0e8a-4716-94ab-581f07c35309",
"name": "02ba10f3-0e8a-4716-94ab-581f07c35309",
"status": "Failed",
"startTime": "2026-10-13T21:54:35.192495222Z",
"endTime": "2026-10-13T22:23:37.481089236Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-fc8dz-zl2sj: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-dpn7kns9h5s6--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-fc8dz-zl2sj to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-fc8dz-zl2sj: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-dpn7kns9h5s6--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-fc8dz-zl2sj to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9256960.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02ba10f3-0e8a-4716-94ab-581f07c35309
correlation ID: 69c597ca-4a5d-45f4-be8b-542aa16b2e4b
client request ID: 15111503-9313-4b39-bc7c-c0bfc7ad586b |
|
| 2h 24m 58s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | westus3 | n/a | passed | #7483 (open) | 0 | n/a | |
| 2h 20m 18s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | centralus | n/a | passed | #6807 (open) | 0 | n/a | |
| 2h 20m 43s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | westus3 | n/a | passed | #7253 (open) | 0 | n/a | |
| 2h 25m 32s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
westus3 | alert, e2e | failed | #7480 (merged) | 2 | e2e (1)ERROR CODE: 429; provider Microsoft.ManagedIdentity Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:108]: failed to get service managed identity
Unexpected error:
<*exported.ResponseError | 0x3788bb56a390>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-02-56/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Request is temporarily throttled because tenant XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-02-56/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
correlation ID: 135296a0-8b8d-4ae4-9dc1-dcc858885005
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_osdisk_encryption.go:108]: failed to get service managed identity
Unexpected error:
<*exported.ResponseError | 0x3788bb56a390>:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-02-56/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
--------------------------------------------------------------------------------
RESPONSE 429: 429 Too Many Requests
ERROR CODE: 429
--------------------------------------------------------------------------------
{
"error": {
"code": "429",
"message": "Request is temporarily throttled because tenant XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has issued too many requests. Retry after 1 seconds."
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-02-56/providers/Microsoft.ManagedIdentity/userAssignedIdentities/service
correlation ID: 135296a0-8b8d-4ae4-9dc1-dcc858885005
client request ID: unavailable
occurredalert (1)alert [hcp] KubePodNotReady fired Show raw failurealert fired 2 time(s) (1 unknown, 1 known) Firing 1: State: Resolved Started: 2026-10-13T21:38:41Z Ended: 2026-10-13T21:44:41Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3189120-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2te4tunmapm3qrg9otufrskbuf4u75vg-h7o1o8e4n0s6g9g", pod="etcd-0", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2te4tunmapm3qrg9otufrskbuf4u75vg-h7o1o8e4n0s6g9g/etcd-0 has been in a non-ready state for longer than 5 minutes. |
|
| 2h 45m 27s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
westus3 | alert, e2e | failed | #7503 (merged) | 3 | e2e (2)ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down pod networks, namespaces; [descendantResources] remaining resources: applydesires, serviceProviderClusters; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x332d8f70c120>:
resource group "rg-hs-pre--pvfm6ns8qtjf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-6j9rv6" in resourcegroup="rg-hs-pre--pvfm6ns8qtjf" to finish deleting: GET https://rp.j4787072.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d2fc814-64dd-4880-8292-2fd063dab951
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d2fc814-64dd-4880-8292-2fd063dab951",
"name": "8d2fc814-64dd-4880-8292-2fd063dab951",
"status": "Failed",
"startTime": "2026-10-13T21:38:32.56326477Z",
"endTime": "2026-10-13T22:02:34.567780855Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4787072.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d2fc814-64dd-4880-8292-2fd063dab951
correlation ID: c238bf11-ea0d-4948-b2da-1eaa7be46473
client request ID: bbe62e16-afcb-49d8-bc81-105532d0b70b
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x332d8f70c120>:
resource group "rg-hs-pre--pvfm6ns8qtjf": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-6j9rv6" in resourcegroup="rg-hs-pre--pvfm6ns8qtjf" to finish deleting: GET https://rp.j4787072.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d2fc814-64dd-4880-8292-2fd063dab951
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d2fc814-64dd-4880-8292-2fd063dab951",
"name": "8d2fc814-64dd-4880-8292-2fd063dab951",
"status": "Failed",
"startTime": "2026-10-13T21:38:32.56326477Z",
"endTime": "2026-10-13T22:02:34.567780855Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4787072.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8d2fc814-64dd-4880-8292-2fd063dab951
correlation ID: c238bf11-ea0d-4948-b2da-1eaa7be46473
client request ID: bbe62e16-afcb-49d8-bc81-105532d0b70b
occurredERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url> Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:123]: failed to create customer resources for y-stream upgrade cluster "cp-ystream-upgrade-4-23-5wrsns"
Unexpected error:
<*fmt.wrapError | 0x9f731ad0ea0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cp-ystream-upgrade-4-23-5wrsns-td2dnf" in resourcegroup="rg-cp-ystream-upgrade-4-23-gn9flc7pzw5b" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cp-ystream-upgrade-4-23-gn9flc7pzw5b/providers/Microsoft.Resources/deployments/customer-infra-cp-ystream-upgrade-4-23-5wrsns-td2dnf/operationStatuses/08584096807961245220
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-mve2kajh3vwg6' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cp-ystream-upgrade-4-23-gn9flc7pzw5b/providers/Microsoft.Resources/deployments/customer-infra-cp-ystream-upgrade-4-23-5wrsns-td2dnf/operationStatuses/08584096807961245220
correlation ID: 547339b0-2310-46d9-9b61-b7c398526e80
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:123]: failed to create customer resources for y-stream upgrade cluster "cp-ystream-upgrade-4-23-5wrsns"
Unexpected error:
<*fmt.wrapError | 0x9f731ad0ea0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cp-ystream-upgrade-4-23-5wrsns-td2dnf" in resourcegroup="rg-cp-ystream-upgrade-4-23-gn9flc7pzw5b" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cp-ystream-upgrade-4-23-gn9flc7pzw5b/providers/Microsoft.Resources/deployments/customer-infra-cp-ystream-upgrade-4-23-5wrsns-td2dnf/operationStatuses/08584096807961245220
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-mve2kajh3vwg6' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cp-ystream-upgrade-4-23-gn9flc7pzw5b/providers/Microsoft.Resources/deployments/customer-infra-cp-ystream-upgrade-4-23-5wrsns-td2dnf/operationStatuses/08584096807961245220
correlation ID: 547339b0-2310-46d9-9b61-b7c398526e80
client request ID: unavailable
occurredalert (1)alert [hcp] KubeDeploymentReplicasMismatch fired Show raw failurealert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T22:15:34Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4787072-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.80:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4787072/providers/microsoft.monitor/accounts/hcps-j4787072", namespace="ocm-arohcpci01-2te530rhjqvehkhg5fuooml9ocotbb8a-j7w4j0s2f4d4g1g", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vkmx4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2te530rhjqvehkhg5fuooml9ocotbb8a-j7w4j0s2f4d4g1g/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-13T22:15:34Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4787072-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.80:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4787072/providers/microsoft.monitor/accounts/hcps-j4787072", namespace="ocm-arohcpci01-2te530rhjqvehkhg5fuooml9ocotbb8a-j7w4j0s2f4d4g1g", pod="arohcp-monitor-kube-state-metrics-7988dc648c-vkmx4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2te530rhjqvehkhg5fuooml9ocotbb8a-j7w4j0s2f4d4g1g/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes. |
|
| 2h 37m 28s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
westus3 | n/a | passed | #7313 (merged) | 0 | n/a | |
| 1h 14m 51s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | unknown | failed | #7504 (open) | 1 | non-artifact-backed failure (no junit artifacts) | |
| 2h 36m 9s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
westus3 | e2e | failed | #7492 (merged) | 1 | e2e (1)failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-12T21:06:23 Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.008s.
Expected success, but got an error:
<*fmt.wrapError | 0x39ffd3acaac0>:
failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-12T21:06:23.2601110Z);
let _endTime = datetime(2026-10-13T21:06:23.2601111Z);
// Extract cluster ID from CS logs using cid field
let _cs_cluster_id = toscalar(
database('ServiceLogs').table('clustersServiceLogs')
| where timestamp between (_startTime .. _endTime)
| where resource_id has 'kusto-logs-tq6r7tbm7n4m'
| where isnotempty(log.cid)
| distinct tostring(log.cid)
| limit 1
);
// Find all Maestro resource IDs for this cluster
let maestro_resource_ids = (
database('ServiceLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has "maestro"
| extend logStr = tostring(log)
| where logStr has _cs_cluster_id
| extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
| where isnotempty(resourceId)
| distinct resourceId
);
// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
union
(
// Maestro logs: Resource bundle operations for this cluster and its associated resources
// Includes: resource creation, updates, deletions, status events, gRPC communications
database('ServiceLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has "maestro"
| extend logStr = tostring(log)
| where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
| project timestamp, container_name, msg = logStr, log
),
(
// Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
// Includes the CR reconciliation logs
database('ServiceLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has "hypershift"
| extend logStr = tostring(log)
| where logStr has _cs_cluster_id
| project timestamp, container_name, msg = logStr, log
),
(
// ACM Agent logs: ManifestWork reconciliation on the managed cluster
// Includes: work agent operations, manifest application, status feedback
database('ServiceLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has "open-cluster-management-agent"
| extend logStr = tostring(log)
| where logStr has _cs_cluster_id
| project timestamp, container_name, msg = logStr, log
)
| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-13T21:06:33.6034532Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558", ActivityId:"91f407a2-0a4c-43e0-acd3-c6532e675296", SubActivityId:"91f407a2-0a4c-43e0-acd3-c6532e675296", ActivityType:"GW.Http.CallContext", ParentActivityId:"91f407a2-0a4c-43e0-acd3-c6532e675296", ActivityStack:"(Activity stack: CRID=KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558 ARID=91f407a2-0a4c-43e0-acd3-c6532e675296 > GW.Http.CallContext/91f407a2-0a4c-43e0-acd3-c6532e675296)"}, IsPermanent:false})
{
msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T21:06:23.2601110Z);\nlet _endTime = datetime(2026-10-13T21:06:23.2601111Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-tq6r7tbm7n4m'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T21:06:33.6034532Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558\", ActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", SubActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityStack:\"(Activity stack: CRID=KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558 ARID=91f407a2-0a4c-43e0-acd3-c6532e675296 > GW.Http.CallContext/91f407a2-0a4c-43e0-acd3-c6532e675296)\"}, IsPermanent:false})",
err: <*errors.joinError | 0x39ffd2546540>{
errs: [
<*fmt.wrapError | 0x39ffd3acaaa0>{
msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T21:06:23.2601110Z);\nlet _endTime = datetime(2026-10-13T21:06:23.2601111Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-tq6r7tbm7n4m'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T21:06:33.6034532Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558\", ActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", SubActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityStack:\"(Activity stack: CRID=KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558 ARID=91f407a2-0a4c-43e0-acd3-c6532e675296 > GW.Http.CallContext/91f407a2-0a4c-43e0-acd3-c6532e675296)\"}, IsPermanent:false})",
err: <*fmt.wrapError | 0x39ffd3acaa80>{
msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T21:06:23.2601110Z);\nlet _endTime = datetime(2026-10-13T21:06:23.2601111Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-tq6r7tbm7n4m'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T21:06:33.6034532Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558\", ActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", SubActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityStack:\"(Activity stack: CRID=KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558 ARID=91f407a2-0a4c-43e0-acd3-c6532e675296 > GW.Http.CallContext/91f407a2-0a4c-43e0-acd3-c6532e675296)\"}, IsPermanent:false})",
err: <*fmt.wrapError | 0x39ffd0047e00>{
msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T21:06:23.2601110Z);\nlet _endTime = datetime(2026-10-13T21:06:23.2601111Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-tq6r7tbm7n4m'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T21:06:33.6034532Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558\", ActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", SubActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityStack:\"(Activity stack: CRID=KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558 ARID=91f407a2-0a4c-43e0-acd3-c6532e675296 > GW.Http.CallContext/91f407a2-0a4c-43e0-acd3-c6532e675296)\"}, IsPermanent:false})",
err: <*fmt.wrapError | 0x39ffd0047dc0>{
msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T21:06:23.2601110Z);\nlet _endTime = datetime(2026-10-13T21:06:23.2601111Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-tq6r7tbm7n4m'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T21:06:33.6034532Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558\", ActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", SubActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityStack:\"(Activity stack: CRID=KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558 ARID=91f407a2-0a4c-43e0-acd3-c6532e675296 > GW.Http.CallContext/91f407a2-0a4c-43e0-acd3-c6532e675296)\"}, IsPermanent:false})",
err: <*v2.OneApiError | 0x39ffd0960100>{
ErrorMessage: {
Code: "LimitsExceeded",
Message: "Request is invalid and cannot be executed.",
Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
Context: {
Timestamp: ...,
ServiceAlias: ...,
MachineName: ...,
ProcessName: ...,
ProcessId: ...,
ThreadId: ...,
ClientRequestId: ...,
ActivityId: ...,
SubActivityId: ...,
ActivityType: ...,
ParentActivityId: ...,
ActivityStack: ...,
},
IsPermanent: false,
},
},
},
},
},
},
],
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:96]: Timed out after 600.008s.
Expected success, but got an error:
<*fmt.wrapError | 0x39ffd3acaac0>:
failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query "detailedInfraOrchestrationLogs" result (KQL: let _startTime = datetime(2026-10-12T21:06:23.2601110Z);
let _endTime = datetime(2026-10-13T21:06:23.2601111Z);
// Extract cluster ID from CS logs using cid field
let _cs_cluster_id = toscalar(
database('ServiceLogs').table('clustersServiceLogs')
| where timestamp between (_startTime .. _endTime)
| where resource_id has 'kusto-logs-tq6r7tbm7n4m'
| where isnotempty(log.cid)
| distinct tostring(log.cid)
| limit 1
);
// Find all Maestro resource IDs for this cluster
let maestro_resource_ids = (
database('ServiceLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has "maestro"
| extend logStr = tostring(log)
| where logStr has _cs_cluster_id
| extend resourceId = extract(@'resourceid[=:"]+([a-f0-9\-]+)', 1, logStr)
| where isnotempty(resourceId)
| distinct resourceId
);
// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent
union
(
// Maestro logs: Resource bundle operations for this cluster and its associated resources
// Includes: resource creation, updates, deletions, status events, gRPC communications
database('ServiceLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has "maestro"
| extend logStr = tostring(log)
| where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)
| project timestamp, container_name, msg = logStr, log
),
(
// Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs
// Includes the CR reconciliation logs
database('ServiceLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has "hypershift"
| extend logStr = tostring(log)
| where logStr has _cs_cluster_id
| project timestamp, container_name, msg = logStr, log
),
(
// ACM Agent logs: ManifestWork reconciliation on the managed cluster
// Includes: work agent operations, manifest application, status feedback
database('ServiceLogs').table('containerLogs')
| where timestamp between (_startTime .. _endTime)
| where namespace_name has "open-cluster-management-agent"
| extend logStr = tostring(log)
| where logStr has _cs_cluster_id
| project timestamp, container_name, msg = logStr, log
)
| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:"LimitsExceeded", Message:"Request is invalid and cannot be executed.", Description:"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..", Type:"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException", Context:v2.ErrorContext{Timestamp:"2026-10-13T21:06:33.6034532Z", ServiceAlias:"HCP-DEV-US-2", MachineName:"", ProcessName:"", ProcessId:0, ThreadId:0, ClientRequestId:"KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558", ActivityId:"91f407a2-0a4c-43e0-acd3-c6532e675296", SubActivityId:"91f407a2-0a4c-43e0-acd3-c6532e675296", ActivityType:"GW.Http.CallContext", ParentActivityId:"91f407a2-0a4c-43e0-acd3-c6532e675296", ActivityStack:"(Activity stack: CRID=KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558 ARID=91f407a2-0a4c-43e0-acd3-c6532e675296 > GW.Http.CallContext/91f407a2-0a4c-43e0-acd3-c6532e675296)"}, IsPermanent:false})
{
msg: "failed to gather logs: failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T21:06:23.2601110Z);\nlet _endTime = datetime(2026-10-13T21:06:23.2601111Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-tq6r7tbm7n4m'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T21:06:33.6034532Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558\", ActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", SubActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityStack:\"(Activity stack: CRID=KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558 ARID=91f407a2-0a4c-43e0-acd3-c6532e675296 > GW.Http.CallContext/91f407a2-0a4c-43e0-acd3-c6532e675296)\"}, IsPermanent:false})",
err: <*errors.joinError | 0x39ffd2546540>{
errs: [
<*fmt.wrapError | 0x39ffd3acaaa0>{
msg: "failed to execute custom logs query: error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T21:06:23.2601110Z);\nlet _endTime = datetime(2026-10-13T21:06:23.2601111Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-tq6r7tbm7n4m'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T21:06:33.6034532Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558\", ActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", SubActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityStack:\"(Activity stack: CRID=KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558 ARID=91f407a2-0a4c-43e0-acd3-c6532e675296 > GW.Http.CallContext/91f407a2-0a4c-43e0-acd3-c6532e675296)\"}, IsPermanent:false})",
err: <*fmt.wrapError | 0x39ffd3acaa80>{
msg: "error during query execution: failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T21:06:23.2601110Z);\nlet _endTime = datetime(2026-10-13T21:06:23.2601111Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-tq6r7tbm7n4m'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T21:06:33.6034532Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558\", ActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", SubActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityStack:\"(Activity stack: CRID=KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558 ARID=91f407a2-0a4c-43e0-acd3-c6532e675296 > GW.Http.CallContext/91f407a2-0a4c-43e0-acd3-c6532e675296)\"}, IsPermanent:false})",
err: <*fmt.wrapError | 0x39ffd0047e00>{
msg: "failed to execute query: failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T21:06:23.2601110Z);\nlet _endTime = datetime(2026-10-13T21:06:23.2601111Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-tq6r7tbm7n4m'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T21:06:33.6034532Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558\", ActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", SubActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityStack:\"(Activity stack: CRID=KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558 ARID=91f407a2-0a4c-43e0-acd3-c6532e675296 > GW.Http.CallContext/91f407a2-0a4c-43e0-acd3-c6532e675296)\"}, IsPermanent:false})",
err: <*fmt.wrapError | 0x39ffd0047dc0>{
msg: "failed to read query \"detailedInfraOrchestrationLogs\" result (KQL: let _startTime = datetime(2026-10-12T21:06:23.2601110Z);\nlet _endTime = datetime(2026-10-13T21:06:23.2601111Z);\n// Extract cluster ID from CS logs using cid field\nlet _cs_cluster_id = toscalar(\n database('ServiceLogs').table('clustersServiceLogs')\n | where timestamp between (_startTime .. _endTime)\n | where resource_id has 'kusto-logs-tq6r7tbm7n4m'\n | where isnotempty(log.cid)\n | distinct tostring(log.cid)\n | limit 1\n);\n// Find all Maestro resource IDs for this cluster\nlet maestro_resource_ids = (\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | extend resourceId = extract(@'resourceid[=:\"]+([a-f0-9\\-]+)', 1, logStr)\n | where isnotempty(resourceId)\n | distinct resourceId\n);\n// Infrastructure & Orchestration Logs - Maestro, Hypershift, and ACM Agent\nunion\n (\n // Maestro logs: Resource bundle operations for this cluster and its associated resources\n // Includes: resource creation, updates, deletions, status events, gRPC communications\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"maestro\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id or logStr has_any (maestro_resource_ids)\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // Hypershift logs: Hypershift operator reconciliation of HostedCluster and NodePool CRs\n // Includes the CR reconciliation logs\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"hypershift\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n ),\n (\n // ACM Agent logs: ManifestWork reconciliation on the managed cluster\n // Includes: work agent operations, manifest application, status feedback\n database('ServiceLogs').table('containerLogs')\n | where timestamp between (_startTime .. _endTime)\n | where namespace_name has \"open-cluster-management-agent\"\n | extend logStr = tostring(log)\n | where logStr has _cs_cluster_id\n | project timestamp, container_name, msg = logStr, log\n )\n| order by timestamp asc): OneApiError(Error=v2.ErrorMessage{Code:\"LimitsExceeded\", Message:\"Request is invalid and cannot be executed.\", Description:\"Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..\", Type:\"Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException\", Context:v2.ErrorContext{Timestamp:\"2026-10-13T21:06:33.6034532Z\", ServiceAlias:\"HCP-DEV-US-2\", MachineName:\"\", ProcessName:\"\", ProcessId:0, ThreadId:0, ClientRequestId:\"KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558\", ActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", SubActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityType:\"GW.Http.CallContext\", ParentActivityId:\"91f407a2-0a4c-43e0-acd3-c6532e675296\", ActivityStack:\"(Activity stack: CRID=KGC.execute;467201e3-34eb-4ddd-9d6d-32112ea40558 ARID=91f407a2-0a4c-43e0-acd3-c6532e675296 > GW.Http.CallContext/91f407a2-0a4c-43e0-acd3-c6532e675296)\"}, IsPermanent:false})",
err: <*v2.OneApiError | 0x39ffd0960100>{
ErrorMessage: {
Code: "LimitsExceeded",
Message: "Request is invalid and cannot be executed.",
Description: "Query execution has exceeded the allowed limits (80DA0003): The results of this query exceed the set limit of 64 MB, so not all records were returned (E_QUERY_RESULT_SET_TOO_LARGE, 0x80DA0003). See https://aka.ms/kustoquerylimits for more information and possible solutions..",
Type: "Kusto.Data.Exceptions.KustoServicePartialQueryFailureLimitsExceededException",
Context: {
Timestamp: ...,
ServiceAlias: ...,
MachineName: ...,
ProcessName: ...,
ProcessId: ...,
ThreadId: ...,
ClientRequestId: ...,
ActivityId: ...,
SubActivityId: ...,
ActivityType: ...,
ParentActivityId: ...,
ActivityStack: ...,
},
IsPermanent: false,
},
},
},
},
},
},
],
},
} |
|
| 51m 20s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
westus3 | provision | failed | #7502 (merged) | 1 | provision (1)ERROR: The subscription of 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' doesn't exist in cloud 'AzureCloud'. Show raw failuretime=2026-10-13T20:10:59.938Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet stamp=2 description="Step swift-vnet\n Kind: Shell\n Command: ./swift-vnet.sh\n" time=2026-10-13T20:11:00.569Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=swift-vnet stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: Failed to load or parse file /tmp/azure-cli-config-533168303/azureProfile.json. It will be overridden by default settings.\nERROR: The subscription of 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' doesn't exist in cloud 'AzureCloud'.\n exit status 1" |
|
| 2h 21m 49s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | alert | failed | #7505 (merged) | 2 | alert (2)alert [svc] FrontendPathLatency fired Show raw failurealert fired 1 time(s)
Firing 1:
State: Resolved
Started: 2026-10-13T20:14:21Z
Ended: 2026-10-13T20:56:21Z
Severity: Sev3
Labels: alertname="FrontendPathLatency", cluster="ci01-j6273152-svc", component="frontend", method="get", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j6273152-svc).alert [svc] userJourneyFrontendLatency6h30m fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T20:38:20Z Ended: 2026-10-13T21:03:20Z Severity: Sev3 Labels: alertname="userJourneyFrontendLatency6h30m", cluster="ci01-j6273152-svc", component="slo", long_window="6h", region="centralus", severity="3", short_window="30m", slo="frontend-latency" Description: More than 6% of Frontend requests on cluster ci01-j6273152-svc exceeded 1s on both the 6h and 30m windows (6x burn of the 1% latency budget). |
|
| 2h 45m 31s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | alert | failed | #7469 (merged) | 2 | alert (2)alert [svc] LeaderElectionLeaseStale fired Show raw failurealert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T20:26:27Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j1682688-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.46:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1682688/providers/microsoft.monitor/accounts/services-j1682688", namespace="klusterlet-2te3djdf1hkgonc8k4i71prc94m2s3ap", pod="arohcp-monitor-kube-state-metrics-7988dc648c-qlgkj", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2te3djdf1hkgonc8k4i71prc94m2s3ap on cluster ci01-j1682688-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. alert [hcp] KubePodNotReady fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T19:33:58Z Ended: 2026-10-13T19:41:56Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j1682688-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2te3dejb287gndjkgcapmskarjrlpqqa-m7o3o1o1z6r6k7r", pod="router-859555d867-955d2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2te3dejb287gndjkgcapmskarjrlpqqa-m7o3o1o1z6r6k7r/router-859555d867-955d2 has been in a non-ready state for longer than 5 minutes. |
|
| 3h 18m 52s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | centralus | alert | failed | #7495 (open) | 3 | alert (3)alert [infra] Cosmos DB Normalized RU Consumption High - <cosmos-account> fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T19:48:13Z Ended: 2026-10-13T20:21:13Z Severity: Sev3 Description: Cosmos DB normalized RU consumption is above 70% averaged over a 15-minute window, evaluated every minute. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units alert [svc] BackendControllerQueueDepthHigh fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T20:03:58Z Ended: 2026-10-13T20:15:58Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6775168-svc", component="backend", name="actualhostedcluster", region="centralus", severity="warning" Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert [infra] Cosmos DB Normalized RU Consumption Burst High - <cosmos-account> fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T19:42:33Z Ended: 2026-10-13T20:09:33Z Severity: Sev3 Description: Cosmos DB normalized RU consumption is above 95% averaged over a 5-minute window, evaluated every minute. This catches short RU-saturation bursts (e.g. mass cluster creation at the start of E2E tests) that exceed provisioned throughput and can lead to 429 throttling before the 10-minute sustained alert fires. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units |
|
| 2h 22m 11s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | n/a | passed | #7249 (merged) | 0 | n/a | |
| 2h 41m 39s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | centralus | alert, e2e | failed | #7477 (closed) | 8 | e2e (7)VerifyHyperShiftIngressCertificate timed out after 10m0 Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:522]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x194b8fb466e0>:
VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x194b8fb466c0>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x194b8fb466a0>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:522]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x194b8fb466e0>:
VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.001s: getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x194b8fb466c0>{
msg: "getting HyperShift ingress secret openshift-ingress/default-ingress-cert: client rate limiter Wait returned an error: context deadline exceeded",
err: <*fmt.wrapError | 0x194b8fb466a0>{
msg: "client rate limiter Wait returned an error: context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
}VerifyHyperShiftIngressCertificate timed out after 10m0 Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:522]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x2b0867292620>:
VerifyHyperShiftIngressCertificate timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x2b08695be1d0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:522]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x2b0867292620>:
VerifyHyperShiftIngressCertificate timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x2b08695be1d0>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}after 4 attempts: 1 releases found in <url> which is not enough for the requested 1 offset Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:134]: failed to resolve configured channel candidate-5.1
Unexpected error:
<*fmt.wrapError | 0xda3675c58e0>:
after 4 attempts: 1 releases found in https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-5.1, which is not enough for the requested 1 offset
{
msg: "after 4 attempts: 1 releases found in https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-5.1, which is not enough for the requested 1 offset",
err: <*errors.errorString | 0xda366e3dab0>{
s: "1 releases found in https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-5.1, which is not enough for the requested 1 offset",
},
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:134]: failed to resolve configured channel candidate-5.1
Unexpected error:
<*fmt.wrapError | 0xda3675c58e0>:
after 4 attempts: 1 releases found in https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-5.1, which is not enough for the requested 1 offset
{
msg: "after 4 attempts: 1 releases found in https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-5.1, which is not enough for the requested 1 offset",
err: <*errors.errorString | 0xda366e3dab0>{
s: "1 releases found in https://api.openshift.com/api/upgrades_info/graph?arch=multi&channel=candidate-5.1, which is not enough for the requested 1 offset",
},
}
occurredVerifyHyperShiftIngressCertificate timed out after 10m0 Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:522]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x14a7efc6800>:
VerifyHyperShiftIngressCertificate timed out after 10m0.002s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.002s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x14a7fffde80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:522]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x14a7efc6800>:
VerifyHyperShiftIngressCertificate timed out after 10m0.002s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.002s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x14a7fffde80>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded" Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-c5gl2m96t572/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-c5gl2m96t572/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: FailedVerifyHyperShiftIngressCertificate timed out after 10m0 Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:522]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x1b7c5f7105e0>:
VerifyHyperShiftIngressCertificate timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x1b7c6068e900>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:522]: HyperShift's ingress certificate secret must exist and be referenced by IngressController/default
Expected success, but got an error:
<*fmt.wrapError | 0x1b7c5f7105e0>:
VerifyHyperShiftIngressCertificate timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret "default-ingress-cert"
{
msg: "VerifyHyperShiftIngressCertificate timed out after 10m0.005s: IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
err: <*errors.errorString | 0x1b7c6068e900>{
s: "IngressController/default references &LocalObjectReference{Name:cluster-ingress-cert,}, expected secret \"default-ingress-cert\"",
},
}clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "<url> Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:405]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0xabc2b9fef0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.11:42173->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.11:42173->172.30.0.10:53: read: connection refused",
}
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert.go:405]: Timed out after 300.005s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
<*errors.errorString | 0xabc2b9fef0>:
clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.11:42173->172.30.0.10:53: read: connection refused
{
s: "clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get \"https://acme-staging-v02.api.letsencrypt.org/directory\": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.11:42173->172.30.0.10:53: read: connection refused",
}alert (1)alert [svc] BackendControllerQueueDepthHigh fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T18:58:36Z Ended: 2026-10-13T19:18:32Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1536384-svc", component="backend", name="clusterresources", region="centralus", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. |
|
| 2h 45m 6s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | westus3 | n/a | passed | #7463 (open) | 0 | n/a | |
| 3h 6m 1s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | westus3 | n/a | passed | #7483 (open) | 0 | n/a | |
| 3h 14m 38s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | alert, e2e | failed | #7248 (merged) | 12 | e2e (8)ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server, ignition-server-proxy]; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-cjhwhd" with version 5.0.0-rc.5 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x2dcd6debe660>:
failed to create HCP cluster cluster-zstream-5-0-cjhwhd: failed waiting for cluster="cluster-zstream-5-0-cjhwhd" in resourcegroup="rg-zstream-upgrade-5-0-zbcrdfvtst44" to finish creating: GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0fa13488-8ecd-48a9-95b8-c369b6566865
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0fa13488-8ecd-48a9-95b8-c369b6566865",
"name": "0fa13488-8ecd-48a9-95b8-c369b6566865",
"status": "Failed",
"startTime": "2026-10-13T18:18:12.815750049Z",
"endTime": "2026-10-13T18:47:19.806209332Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0fa13488-8ecd-48a9-95b8-c369b6566865
correlation ID: d259ddfa-e906-4fbb-974b-3c977af82a32
client request ID: be895625-b480-4da8-a968-9e02baae7cef
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-5-0-cjhwhd" with version 5.0.0-rc.5 on candidate channel
Unexpected error:
<*fmt.wrapError | 0x2dcd6debe660>:
failed to create HCP cluster cluster-zstream-5-0-cjhwhd: failed waiting for cluster="cluster-zstream-5-0-cjhwhd" in resourcegroup="rg-zstream-upgrade-5-0-zbcrdfvtst44" to finish creating: GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0fa13488-8ecd-48a9-95b8-c369b6566865
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0fa13488-8ecd-48a9-95b8-c369b6566865",
"name": "0fa13488-8ecd-48a9-95b8-c369b6566865",
"status": "Failed",
"startTime": "2026-10-13T18:18:12.815750049Z",
"endTime": "2026-10-13T18:47:19.806209332Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0fa13488-8ecd-48a9-95b8-c369b6566865
correlation ID: d259ddfa-e906-4fbb-974b-3c977af82a32
client request ID: be895625-b480-4da8-a968-9e02baae7cef
occurredERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-4x5sn82fcphp/cluster-candidate-4-22-zrq2s6 should provision
Unexpected error:
<*fmt.wrapError | 0x19aa63530420>:
failed to create HCP cluster cluster-candidate-4-22-zrq2s6: failed waiting for cluster="cluster-candidate-4-22-zrq2s6" in resourcegroup="rg-candidate-4-22-4x5sn82fcphp" to finish creating: GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a9c2b23d-2530-4218-9438-90a72b3ed5c3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a9c2b23d-2530-4218-9438-90a72b3ed5c3",
"name": "a9c2b23d-2530-4218-9438-90a72b3ed5c3",
"status": "Failed",
"startTime": "2026-10-13T18:18:33.062727378Z",
"endTime": "2026-10-13T18:47:39.762455137Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 2 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a9c2b23d-2530-4218-9438-90a72b3ed5c3
correlation ID: 0a0f8159-fd50-4585-9f5a-4fe4a3b75abe
client request ID: 2d0d801d-1f09-4e1e-b3d0-c47d56f382dc
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-4x5sn82fcphp/cluster-candidate-4-22-zrq2s6 should provision
Unexpected error:
<*fmt.wrapError | 0x19aa63530420>:
failed to create HCP cluster cluster-candidate-4-22-zrq2s6: failed waiting for cluster="cluster-candidate-4-22-zrq2s6" in resourcegroup="rg-candidate-4-22-4x5sn82fcphp" to finish creating: GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a9c2b23d-2530-4218-9438-90a72b3ed5c3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a9c2b23d-2530-4218-9438-90a72b3ed5c3",
"name": "a9c2b23d-2530-4218-9438-90a72b3ed5c3",
"status": "Failed",
"startTime": "2026-10-13T18:18:33.062727378Z",
"endTime": "2026-10-13T18:47:39.762455137Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 2 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a9c2b23d-2530-4218-9438-90a72b3ed5c3
correlation ID: 0a0f8159-fd50-4585-9f5a-4fe4a3b75abe
client request ID: 2d0d801d-1f09-4e1e-b3d0-c47d56f382dc
occurredERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-qcrtxl" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x124373e0e100>:
failed to create HCP cluster cp-ystream-upgrade-4-23-qcrtxl: failed waiting for cluster="cp-ystream-upgrade-4-23-qcrtxl" in resourcegroup="rg-cp-ystream-upgrade-4-23-78kjmz9gspg6" to finish creating: GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d1a5e31e-a45c-4250-99d1-c48ac813ea28
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d1a5e31e-a45c-4250-99d1-c48ac813ea28",
"name": "d1a5e31e-a45c-4250-99d1-c48ac813ea28",
"status": "Failed",
"startTime": "2026-10-13T18:18:30.082143101Z",
"endTime": "2026-10-13T18:47:39.605099419Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d1a5e31e-a45c-4250-99d1-c48ac813ea28
correlation ID: 361fa06a-5b9c-42df-b411-6ad9494cda34
client request ID: 5835d195-02fb-460c-b9e4-3f1748f27ec9
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-qcrtxl" at install version 4.22
Unexpected error:
<*fmt.wrapError | 0x124373e0e100>:
failed to create HCP cluster cp-ystream-upgrade-4-23-qcrtxl: failed waiting for cluster="cp-ystream-upgrade-4-23-qcrtxl" in resourcegroup="rg-cp-ystream-upgrade-4-23-78kjmz9gspg6" to finish creating: GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d1a5e31e-a45c-4250-99d1-c48ac813ea28
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d1a5e31e-a45c-4250-99d1-c48ac813ea28",
"name": "d1a5e31e-a45c-4250-99d1-c48ac813ea28",
"status": "Failed",
"startTime": "2026-10-13T18:18:30.082143101Z",
"endTime": "2026-10-13T18:47:39.605099419Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d1a5e31e-a45c-4250-99d1-c48ac813ea28
correlation ID: 361fa06a-5b9c-42df-b411-6ad9494cda34
client request ID: 5835d195-02fb-460c-b9e4-3f1748f27ec9
occurredERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, router]; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0x384da2442800>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-65x7ldfvpfrb" to finish creating: GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8d7260b8-3447-49f3-b1d1-85b028cd4875
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8d7260b8-3447-49f3-b1d1-85b028cd4875",
"name": "8d7260b8-3447-49f3-b1d1-85b028cd4875",
"status": "Failed",
"startTime": "2026-10-13T18:19:32.972571872Z",
"endTime": "2026-10-13T18:48:39.815141519Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8d7260b8-3447-49f3-b1d1-85b028cd4875
correlation ID: 7e9c28a1-2d97-4241-837c-65dbd7be5ebd
client request ID: 76ea1b85-c87c-48c0-b8b1-76222a78e914
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
<*fmt.wrapError | 0x384da2442800>:
failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-65x7ldfvpfrb" to finish creating: GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8d7260b8-3447-49f3-b1d1-85b028cd4875
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8d7260b8-3447-49f3-b1d1-85b028cd4875",
"name": "8d7260b8-3447-49f3-b1d1-85b028cd4875",
"status": "Failed",
"startTime": "2026-10-13T18:19:32.972571872Z",
"endTime": "2026-10-13T18:48:39.815141519Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, router deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8d7260b8-3447-49f3-b1d1-85b028cd4875
correlation ID: 7e9c28a1-2d97-4241-837c-65dbd7be5ebd
client request ID: 76ea1b85-c87c-48c0-b8b1-76222a78e914
occurredERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider, ignition-server, ignition-server-proxy]; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-dllvsm" at install version 4.20
Unexpected error:
<*fmt.wrapError | 0x28758e830200>:
failed to create HCP cluster cp-ystream-upgrade-4-21-dllvsm: failed waiting for cluster="cp-ystream-upgrade-4-21-dllvsm" in resourcegroup="rg-cp-ystream-upgrade-4-21-kqbmkzfdlvpv" to finish creating: GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/96ec3c05-7e89-48a9-8588-f769d1d82955
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/96ec3c05-7e89-48a9-8588-f769d1d82955",
"name": "96ec3c05-7e89-48a9-8588-f769d1d82955",
"status": "Failed",
"startTime": "2026-10-13T18:18:30.340236179Z",
"endTime": "2026-10-13T18:47:39.7699444Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/96ec3c05-7e89-48a9-8588-f769d1d82955
correlation ID: 8e81f893-e896-41de-bfb5-ecaf7ea09f96
client request ID: 8d55acf1-8d30-4678-b2f6-7eed037af357
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-dllvsm" at install version 4.20
Unexpected error:
<*fmt.wrapError | 0x28758e830200>:
failed to create HCP cluster cp-ystream-upgrade-4-21-dllvsm: failed waiting for cluster="cp-ystream-upgrade-4-21-dllvsm" in resourcegroup="rg-cp-ystream-upgrade-4-21-kqbmkzfdlvpv" to finish creating: GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/96ec3c05-7e89-48a9-8588-f769d1d82955
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/96ec3c05-7e89-48a9-8588-f769d1d82955",
"name": "96ec3c05-7e89-48a9-8588-f769d1d82955",
"status": "Failed",
"startTime": "2026-10-13T18:18:30.340236179Z",
"endTime": "2026-10-13T18:47:39.7699444Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, ignition-server deployment has 1 unavailable replicas, ignition-server-proxy deployment has 1 unavailable replicas]"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/96ec3c05-7e89-48a9-8588-f769d1d82955
correlation ID: 8e81f893-e896-41de-bfb5-ecaf7ea09f96
client request ID: 8d55acf1-8d30-4678-b2f6-7eed037af357
occurredERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:125]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x178e97a22840>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-w76dhl6mtjzd" to finish creating: GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/235e4bf2-54cf-43d0-ab9b-65d9a733710d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/235e4bf2-54cf-43d0-ab9b-65d9a733710d",
"name": "235e4bf2-54cf-43d0-ab9b-65d9a733710d",
"status": "Failed",
"startTime": "2026-10-13T18:18:43.086930646Z",
"endTime": "2026-10-13T18:47:49.831018491Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/235e4bf2-54cf-43d0-ab9b-65d9a733710d
correlation ID: 077ff34a-ea54-4d78-afd3-f4054328a8bd
client request ID: 612adb31-4506-4736-9005-28fa95bb228f
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:125]: failed to create HCP cluster
Unexpected error:
<*fmt.wrapError | 0x178e97a22840>:
failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-w76dhl6mtjzd" to finish creating: GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/235e4bf2-54cf-43d0-ab9b-65d9a733710d
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/235e4bf2-54cf-43d0-ab9b-65d9a733710d",
"name": "235e4bf2-54cf-43d0-ab9b-65d9a733710d",
"status": "Failed",
"startTime": "2026-10-13T18:18:43.086930646Z",
"endTime": "2026-10-13T18:47:49.831018491Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/235e4bf2-54cf-43d0-ab9b-65d9a733710d
correlation ID: 077ff34a-ea54-4d78-afd3-f4054328a8bd
client request ID: 612adb31-4506-4736-9005-28fa95bb228f
occurredcluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded" Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-pgbspldcrvd7/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-pgbspldcrvd7/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: FailedERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-mcx8fpbfcpnk/cluster-candidate-5-0-vlgchl should provision
Unexpected error:
<*fmt.wrapError | 0x109506636520>:
failed to create HCP cluster cluster-candidate-5-0-vlgchl: failed waiting for cluster="cluster-candidate-5-0-vlgchl" in resourcegroup="rg-candidate-5-0-mcx8fpbfcpnk" to finish creating: GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1b967a2-e91b-49c3-83e2-d3b8a97a8f24
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1b967a2-e91b-49c3-83e2-d3b8a97a8f24",
"name": "a1b967a2-e91b-49c3-83e2-d3b8a97a8f24",
"status": "Failed",
"startTime": "2026-10-13T18:18:32.401074593Z",
"endTime": "2026-10-13T18:47:39.748542315Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1b967a2-e91b-49c3-83e2-d3b8a97a8f24
correlation ID: 8db3f564-6943-4e86-ae53-344e19b24905
client request ID: 395fee37-68bf-4f23-8767-6c77bcf873de
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-mcx8fpbfcpnk/cluster-candidate-5-0-vlgchl should provision
Unexpected error:
<*fmt.wrapError | 0x109506636520>:
failed to create HCP cluster cluster-candidate-5-0-vlgchl: failed waiting for cluster="cluster-candidate-5-0-vlgchl" in resourcegroup="rg-candidate-5-0-mcx8fpbfcpnk" to finish creating: GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1b967a2-e91b-49c3-83e2-d3b8a97a8f24
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1b967a2-e91b-49c3-83e2-d3b8a97a8f24",
"name": "a1b967a2-e91b-49c3-83e2-d3b8a97a8f24",
"status": "Failed",
"startTime": "2026-10-13T18:18:32.401074593Z",
"endTime": "2026-10-13T18:47:39.748542315Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j2423040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a1b967a2-e91b-49c3-83e2-d3b8a97a8f24
correlation ID: 8db3f564-6943-4e86-ae53-344e19b24905
client request ID: 395fee37-68bf-4f23-8767-6c77bcf873de
occurredalert (4)alert [hcp] KubePodNotReady fired Show raw failurealert fired 14 time(s) (4 unknown, 10 known) Firing 1: State: Resolved Started: 2026-10-13T18:27:25Z Ended: 2026-10-13T19:00:28Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2te2g0ue1d1l7frlbcjrgu6374q9f127-ingress-cert-le", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2te2g0ue1d1l7frlbcjrgu6374q9f127-ingress-cert-le/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-10-13T18:28:35Z Ended: 2026-10-13T19:00:28Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2te2ftrfr2171c9vgv1tegaimecvl6vj-g6a2f9w2l2r4x0i", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2te2ftrfr2171c9vgv1tegaimecvl6vj-g6a2f9w2l2r4x0i/etcd-2 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-10-13T18:28:36Z Ended: 2026-10-13T19:00:28Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2te2fuhn0rfvdiv69jgohnlv63fgjjka-q9h0p7f7k8t5e0e", pod="etcd-0", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2te2fuhn0rfvdiv69jgohnlv63fgjjka-q9h0p7f7k8t5e0e/etcd-0 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-10-13T18:28:36Z Ended: 2026-10-13T19:00:28Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2te2fpcp41do03q2ch5ajhtin6o2onk6-n6k1o6x0q6g7e9r", pod="etcd-2", region="centralus", severity="warning" Description: Pod ocm-arohcpci01-2te2fpcp41do03q2ch5ajhtin6o2onk6-n6k1o6x0q6g7e9r/etcd-2 has been in a non-ready state for longer than 5 minutes. alert [hcp] KubeDeploymentReplicasMismatch fired Show raw failurealert fired 5 time(s) Firing 1: State: Resolved Started: 2026-10-13T18:51:30Z Ended: 2026-10-13T19:06:24Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/hcps-j2423040", namespace="ocm-arohcpci01-2te2g0ue1d1l7frlbcjrgu6374q9f127-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2te2g0ue1d1l7frlbcjrgu6374q9f127-ingress-cert-le/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-13T18:52:27Z Ended: 2026-10-13T19:07:25Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/hcps-j2423040", namespace="ocm-arohcpci01-2te2g0ue1d1l7frlbcjrgu6374q9f127-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2te2g0ue1d1l7frlbcjrgu6374q9f127-ingress-cert-le/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 3: State: Resolved Started: 2026-10-13T18:52:27Z Ended: 2026-10-13T19:06:24Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/hcps-j2423040", namespace="ocm-arohcpci01-2te2fpcp41do03q2ch5ajhtin6o2onk6-n6k1o6x0q6g7e9r", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2te2fpcp41do03q2ch5ajhtin6o2onk6-n6k1o6x0q6g7e9r/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 4: State: Resolved Started: 2026-10-13T18:53:28Z Ended: 2026-10-13T19:05:28Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/hcps-j2423040", namespace="ocm-arohcpci01-2te2ftrfr2171c9vgv1tegaimecvl6vj-g6a2f9w2l2r4x0i", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2te2ftrfr2171c9vgv1tegaimecvl6vj-g6a2f9w2l2r4x0i/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 5: State: Resolved Started: 2026-10-13T18:53:28Z Ended: 2026-10-13T19:19:31Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/hcps-j2423040", namespace="ocm-arohcpci01-2te2fuhn0rfvdiv69jgohnlv63fgjjka-q9h0p7f7k8t5e0e", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2te2fuhn0rfvdiv69jgohnlv63fgjjka-q9h0p7f7k8t5e0e/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert [hcp] KubeStatefulSetReplicasMismatch fired Show raw failurealert fired 8 time(s) Firing 1: State: Resolved Started: 2026-10-13T18:37:29Z Ended: 2026-10-13T18:55:26Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/hcps-j2423040", namespace="ocm-arohcpci01-2te2g0ue1d1l7frlbcjrgu6374q9f127-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2te2g0ue1d1l7frlbcjrgu6374q9f127-ingress-cert-le/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-13T18:37:29Z Ended: 2026-10-13T18:54:27Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/hcps-j2423040", namespace="ocm-arohcpci01-2te2g0ue1d1l7frlbcjrgu6374q9f127-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2te2g0ue1d1l7frlbcjrgu6374q9f127-ingress-cert-le/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 3: State: Resolved Started: 2026-10-13T18:38:27Z Ended: 2026-10-13T18:55:26Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/hcps-j2423040", namespace="ocm-arohcpci01-2te2fpcp41do03q2ch5ajhtin6o2onk6-n6k1o6x0q6g7e9r", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2te2fpcp41do03q2ch5ajhtin6o2onk6-n6k1o6x0q6g7e9r/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 4: State: Resolved Started: 2026-10-13T18:38:27Z Ended: 2026-10-13T18:55:27Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/hcps-j2423040", namespace="ocm-arohcpci01-2te2fuhn0rfvdiv69jgohnlv63fgjjka-q9h0p7f7k8t5e0e", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2te2fuhn0rfvdiv69jgohnlv63fgjjka-q9h0p7f7k8t5e0e/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 5: State: Resolved Started: 2026-10-13T18:38:27Z Ended: 2026-10-13T18:54:27Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/hcps-j2423040", namespace="ocm-arohcpci01-2te2ftrfr2171c9vgv1tegaimecvl6vj-g6a2f9w2l2r4x0i", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2te2ftrfr2171c9vgv1tegaimecvl6vj-g6a2f9w2l2r4x0i/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 6: State: Resolved Started: 2026-10-13T18:38:28Z Ended: 2026-10-13T18:54:27Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/hcps-j2423040", namespace="ocm-arohcpci01-2te2fuhn0rfvdiv69jgohnlv63fgjjka-q9h0p7f7k8t5e0e", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2te2fuhn0rfvdiv69jgohnlv63fgjjka-q9h0p7f7k8t5e0e/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 7: State: Resolved Started: 2026-10-13T18:38:28Z Ended: 2026-10-13T18:55:26Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/hcps-j2423040", namespace="ocm-arohcpci01-2te2ftrfr2171c9vgv1tegaimecvl6vj-g6a2f9w2l2r4x0i", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2te2ftrfr2171c9vgv1tegaimecvl6vj-g6a2f9w2l2r4x0i/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 8: State: Resolved Started: 2026-10-13T18:38:28Z Ended: 2026-10-13T18:54:27Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/hcps-j2423040", namespace="ocm-arohcpci01-2te2fpcp41do03q2ch5ajhtin6o2onk6-n6k1o6x0q6g7e9r", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2te2fpcp41do03q2ch5ajhtin6o2onk6-n6k1o6x0q6g7e9r/etcd has not matched the expected number of replicas for longer than 15 minutes. alert [svc] KubeDeploymentReplicasMismatch fired Show raw failurealert fired 12 time(s) Firing 1: State: Resolved Started: 2026-10-13T18:51:01Z Ended: 2026-10-13T18:58:00Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/services-j2423040", namespace="klusterlet-2te2g0ue1d1l7frlbcjrgu6374q9f127", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2te2g0ue1d1l7frlbcjrgu6374q9f127/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-13T18:51:02Z Ended: 2026-10-13T18:59:02Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/services-j2423040", namespace="klusterlet-2te2g0ue1d1l7frlbcjrgu6374q9f127", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2te2g0ue1d1l7frlbcjrgu6374q9f127/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes. Firing 3: State: Resolved Started: 2026-10-13T18:51:02Z Ended: 2026-10-13T18:59:03Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/services-j2423040", namespace="klusterlet-2te2fpcp41do03q2ch5ajhtin6o2onk6", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2te2fpcp41do03q2ch5ajhtin6o2onk6/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes. Firing 4: State: Resolved Started: 2026-10-13T18:51:02Z Ended: 2026-10-13T18:58:00Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/services-j2423040", namespace="klusterlet-2te2g0ue1d1l7frlbcjrgu6374q9f127", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2te2g0ue1d1l7frlbcjrgu6374q9f127/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 5: State: Resolved Started: 2026-10-13T18:52:01Z Ended: 2026-10-13T18:59:02Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/services-j2423040", namespace="klusterlet-2te2ftrfr2171c9vgv1tegaimecvl6vj", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2te2ftrfr2171c9vgv1tegaimecvl6vj/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes. Firing 6: State: Resolved Started: 2026-10-13T18:52:01Z Ended: 2026-10-13T18:59:02Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/services-j2423040", namespace="klusterlet-2te2ftrfr2171c9vgv1tegaimecvl6vj", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2te2ftrfr2171c9vgv1tegaimecvl6vj/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes. Firing 7: State: Resolved Started: 2026-10-13T18:52:01Z Ended: 2026-10-13T18:58:01Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/services-j2423040", namespace="klusterlet-2te2ftrfr2171c9vgv1tegaimecvl6vj", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2te2ftrfr2171c9vgv1tegaimecvl6vj/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 8: State: Resolved Started: 2026-10-13T18:52:01Z Ended: 2026-10-13T18:58:01Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/services-j2423040", namespace="klusterlet-2te2ftrfr2171c9vgv1tegaimecvl6vj", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2te2ftrfr2171c9vgv1tegaimecvl6vj/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 9: State: Resolved Started: 2026-10-13T18:52:01Z Ended: 2026-10-13T18:59:02Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/services-j2423040", namespace="klusterlet-2te2g0ue1d1l7frlbcjrgu6374q9f127", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2te2g0ue1d1l7frlbcjrgu6374q9f127/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes. Firing 10: State: Resolved Started: 2026-10-13T18:52:58Z Ended: 2026-10-13T18:59:02Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/services-j2423040", namespace="klusterlet-2te2fuhn0rfvdiv69jgohnlv63fgjjka", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2te2fuhn0rfvdiv69jgohnlv63fgjjka/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes. Firing 11: State: Resolved Started: 2026-10-13T18:52:58Z Ended: 2026-10-13T18:59:02Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/services-j2423040", namespace="klusterlet-2te2fuhn0rfvdiv69jgohnlv63fgjjka", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2te2fuhn0rfvdiv69jgohnlv63fgjjka/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes. Firing 12: State: Resolved Started: 2026-10-13T18:54:01Z Ended: 2026-10-13T19:01:00Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2423040-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.82:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2423040/providers/microsoft.monitor/accounts/services-j2423040", namespace="klusterlet-2te2gdnt0p29rhn6u0lod1kmga0s4ejn", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mgp99", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2te2gdnt0p29rhn6u0lod1kmga0s4ejn/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes. |
|
| 16m 8s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | unknown | failed | #7477 (closed) | 1 | non-artifact-backed failure (no junit artifacts) | |
| 3h 13m 6s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | centralus | alert | failed | #7504 (open) | 1 | alert (1)alert [svc] LeaderElectionLeaseStale fired Show raw failurealert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T19:14:04Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j9671808-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.101:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9671808/providers/microsoft.monitor/accounts/services-j9671808", namespace="klusterlet-2te2bel6rgtolfhffo3974figin61sko", pod="arohcp-monitor-kube-state-metrics-7988dc648c-z7lt7", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2te2bel6rgtolfhffo3974figin61sko on cluster ci01-j9671808-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. |
|
| 2h 27m 47s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
westus3 | alert, provision | failed | #7502 (merged) | 4 | provision (1)failed to roll out Helm release: resource Deployment/maestro/maestro not ready. status: Failed, message: Progress deadline exceeded Show raw failuretime=2026-10-13T18:29:04.438Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n" time=2026-10-13T18:29:04.984Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy values="map[azure:map[clientId:e73242a2-1d48-4914-b494-303251483672 cloudEnvironmentName:AzurePublicCloud tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] backend:map[namespace:aro-hcp serviceAccount:backend] broker:map[host:arohcp-ci01-maestro-j4029440.westus3-1.ts.eventgrid.azure.net port:8883] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] credsKeyVault:map[name:aro-hcp-dev-svc-kv secret:maestro-server-j4029440] database:map[azureDb:false containerizedDbImage:docker.io/library/postgres:14.2 containerizedDbPvcCapacity:512Mi debug:false host:maestro-db maxOpenConnections:50 name:maestro password:TheBlurstOfTimes port:5432 user:maestro-server] deployment:map[replicas:2 resources:map[limits:map[memory:unlimited] requests:map[cpu:200m memory:512Mi]] zoneCount:3] fleet:map[namespace:fleet serviceAccount:fleet] image:map[digest:sha256:ccc6c4381bbbc6a7af65d4c3886e20baea76f7bb95ea6048b33ee31e8046e5cb registry:arohcpsvcdev.azurecr.io repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro] istio:map[restrictIngress:true] maestro:map[glog_v:2 grpcBindPort:8090 healthCheckBindPort:8083 httpBindPort:8000 httpReadTimeout:60s httpWriteTimeout:60s labelMetricsInclusionDuration:168h serviceAccount:maestro] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:redhat-user-workloads/maestro-rhtap-tenant/maestro/maestro:pull workloadIdentityClientId:1176baba-6cca-46e9-86bb-3a9c587b6d0c workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX]]" time=2026-10-13T18:29:04.984Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy time=2026-10-13T18:29:05.371Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy namespace=maestro time=2026-10-13T18:29:05.621Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy dryRun=false time=2026-10-13T18:29:05.621Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy time=2026-10-13T18:29:05.751Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy time=2026-10-13T18:39:13.113Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy err="resource Deployment/maestro/maestro not ready. status: Failed, message: Progress deadline exceeded" time=2026-10-13T18:39:13.113Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Maestro.Server resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/maestro/maestro not ready. status: Failed, message: Progress deadline exceeded" alert (3)alert [svc] MaestroAgentMetricsProxyDown fired Show raw failurealert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-13T18:24:20Z Ended: 2026-10-13T18:31:19Z Severity: Sev3 Labels: alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j4029440-mgmt-1", component="monitoring-infrastructure", region="westus3", severity="3" Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing. Firing 2: State: Resolved Started: 2026-10-13T18:24:20Z Ended: 2026-10-13T18:31:19Z Severity: Sev3 Labels: alertname="MaestroAgentMetricsProxyDown", cluster="ci01-j4029440-mgmt-2", component="monitoring-infrastructure", region="westus3", severity="3" Description: Prometheus has not scraped maestro-agent /metrics through the metrics-proxy sidecar (namespace=maestro, endpoint=metrics) for 10 minutes, while a kube-state-metrics scrape series still exists for this management cluster. Maestro agent metrics and any alerts that depend on them may be missing. alert [svc] MaestroServerNoReadyReplicas fired Show raw failurealert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T18:40:06Z Severity: Sev2 Labels: alertname="MaestroServerNoReadyReplicas", cluster="ci01-j4029440-svc", component="maestro", container="kube-state-metrics", deployment="maestro", endpoint="http", environment="ci01", instance="10.128.64.59:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4029440/providers/microsoft.monitor/accounts/services-j4029440", namespace="maestro", pod="arohcp-monitor-kube-state-metrics-79b88f5699-k5lj9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="critical" Description: No maestro-server replicas have been Ready for 10 minutes. Readiness is gated on Postgres connectivity, so maestro is likely unable to reach its database; Clusters Service will get 503 and HCP cluster provisioning will stall. Firing 2: State: Fired (not resolved) Started: 2026-10-13T18:40:06Z Severity: Sev2 Labels: alertname="MaestroServerNoReadyReplicas", cluster="ci01-j4029440-svc", component="maestro", container="kube-state-metrics", deployment="maestro", endpoint="http", environment="ci01", instance="10.128.64.59:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4029440/providers/microsoft.monitor/accounts/services-j4029440", namespace="maestro", pod="arohcp-monitor-kube-state-metrics-79b88f5699-k5lj9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="critical" Description: No maestro-server replicas have been Ready for 10 minutes. Readiness is gated on Postgres connectivity, so maestro is likely unable to reach its database; Clusters Service will get 503 and HCP cluster provisioning will stall. alert [svc] KubePodNotReady fired Show raw failurealert fired 17 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T18:32:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-hllvl", region="westus3", severity="warning" Description: Pod kube-system/ama-metrics-node-hllvl has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-13T18:32:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-14034537-vmss000000-xc7fq", region="westus3", severity="warning" Description: Pod kube-system/eraser-aks-system-14034537-vmss000000-xc7fq has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-10-13T18:32:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-4k67c", region="westus3", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-4k67c has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-10-13T18:32:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-czf22", region="westus3", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-czf22 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Fired (not resolved) Started: 2026-10-13T18:32:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-14034537-vmss000002-9swqr", region="westus3", severity="warning" Description: Pod kube-system/eraser-aks-system-14034537-vmss000002-9swqr has been in a non-ready state for longer than 5 minutes. Firing 6: State: Fired (not resolved) Started: 2026-10-13T18:32:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="hubble-relay-58594b5dd7-nnl6j", region="westus3", severity="warning" Description: Pod kube-system/hubble-relay-58594b5dd7-nnl6j has been in a non-ready state for longer than 5 minutes. Firing 7: State: Fired (not resolved) Started: 2026-10-13T18:32:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="eraser-aks-system-14034537-vmss000001-8tbkl", region="westus3", severity="warning" Description: Pod kube-system/eraser-aks-system-14034537-vmss000001-8tbkl has been in a non-ready state for longer than 5 minutes. Firing 8: State: Fired (not resolved) Started: 2026-10-13T18:32:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="aks-secrets-store-csi-driver-jvbz8", region="westus3", severity="warning" Description: Pod kube-system/aks-secrets-store-csi-driver-jvbz8 has been in a non-ready state for longer than 5 minutes. Firing 9: State: Fired (not resolved) Started: 2026-10-13T18:32:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-nn274", region="westus3", severity="warning" Description: Pod kube-system/ama-metrics-node-nn274 has been in a non-ready state for longer than 5 minutes. Firing 10: State: Fired (not resolved) Started: 2026-10-13T18:32:47Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="kube-system", pod="ama-metrics-node-7rvmk", region="westus3", severity="warning" Description: Pod kube-system/ama-metrics-node-7rvmk has been in a non-ready state for longer than 5 minutes. Firing 11: State: Fired (not resolved) Started: 2026-10-13T18:34:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-c67c4f659-qcdm6", region="westus3", severity="warning" Description: Pod clusters-service/clusters-service-c67c4f659-qcdm6 has been in a non-ready state for longer than 5 minutes. Firing 12: State: Fired (not resolved) Started: 2026-10-13T18:34:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="maestro", pod="maestro-7dbd665f4c-8rntv", region="westus3", severity="warning" Description: Pod maestro/maestro-7dbd665f4c-8rntv has been in a non-ready state for longer than 5 minutes. Firing 13: State: Fired (not resolved) Started: 2026-10-13T18:34:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-c67c4f659-c7v65", region="westus3", severity="warning" Description: Pod clusters-service/clusters-service-c67c4f659-c7v65 has been in a non-ready state for longer than 5 minutes. Firing 14: State: Fired (not resolved) Started: 2026-10-13T18:34:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="maestro", pod="maestro-7dbd665f4c-474rm", region="westus3", severity="warning" Description: Pod maestro/maestro-7dbd665f4c-474rm has been in a non-ready state for longer than 5 minutes. Firing 15: State: Fired (not resolved) Started: 2026-10-13T18:34:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="clusters-service", pod="clusters-service-c67c4f659-4zmsg", region="westus3", severity="warning" Description: Pod clusters-service/clusters-service-c67c4f659-4zmsg has been in a non-ready state for longer than 5 minutes. Firing 16: State: Fired (not resolved) Started: 2026-10-13T18:34:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-698d94bbf-pwqx5", region="westus3", severity="warning" Description: Pod sessiongate/sessiongate-698d94bbf-pwqx5 has been in a non-ready state for longer than 5 minutes. Firing 17: State: Fired (not resolved) Started: 2026-10-13T18:34:45Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4029440-svc", component="kubernetes-infrastructure", namespace="sessiongate", pod="sessiongate-698d94bbf-7crpb", region="westus3", severity="warning" Description: Pod sessiongate/sessiongate-698d94bbf-7crpb has been in a non-ready state for longer than 5 minutes. |
|
| 3h 2m 58s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | n/a | passed | #7397 (merged) | 0 | n/a | |
| 3h 47m 19s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | alert | failed | #7488 (merged) | 1 | alert (1)alert [svc] LeaderElectionLeaseStale fired Show raw failurealert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T19:29:44Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j1296000-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.78:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1296000/providers/microsoft.monitor/accounts/services-j1296000", namespace="klusterlet-2te2i26cvbt70m1icpl3fmd77s52ciq8", pod="arohcp-monitor-kube-state-metrics-7988dc648c-86rpv", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2te2i26cvbt70m1icpl3fmd77s52ciq8 on cluster ci01-j1296000-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. |
|
| 3h 18m 51s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
westus3 | alert | failed | #7162 (merged) | 1 | alert (1)alert [svc] LeaderElectionLeaseStale fired Show raw failurealert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T18:37:57Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j2195456-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.244:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2195456/providers/microsoft.monitor/accounts/services-j2195456", namespace="klusterlet-2te22fpnsean3o8qbccv619h2micg5vv", pod="arohcp-monitor-kube-state-metrics-7988dc648c-hcksn", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2te22fpnsean3o8qbccv619h2micg5vv on cluster ci01-j2195456-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. |
|
| 2h 25m 38s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
westus3 | n/a | passed | #7484 (merged) | 0 | n/a | |
| 2h 16m 45s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | n/a | passed | #7494 (merged) | 0 | n/a | |
| 2h 21m 58s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | e2e | failed | #7313 (merged) | 2 | e2e (2)failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x1261114a4180>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-sd447nrw2b6l" to finish creating: GET https://rp.j0059904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7256f4ec-b394-462d-a65c-c0c44cf424ed
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7256f4ec-b394-462d-a65c-c0c44cf424ed",
"name": "7256f4ec-b394-462d-a65c-c0c44cf424ed",
"status": "Failed",
"startTime": "2026-10-13T16:58:45.675244209Z",
"endTime": "2026-10-13T17:27:48.586418683Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o0z7z4b2j5v2y3n-np-delete-main-bbsn2-mr9vl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-sd447nrw2b6l--managed/providers/Microsoft.Compute/virtualMachines/o0z7z4b2j5v2y3n-np-delete-main-bbsn2-mr9vl to exist\nMachine o0z7z4b2j5v2y3n-np-delete-main-bbsn2-ms6dr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-sd447nrw2b6l--managed/providers/Microsoft.Compute/virtualMachines/o0z7z4b2j5v2y3n-np-delete-main-bbsn2-ms6dr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o0z7z4b2j5v2y3n-np-delete-main-bbsn2-mr9vl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-sd447nrw2b6l--managed/providers/Microsoft.Compute/virtualMachines/o0z7z4b2j5v2y3n-np-delete-main-bbsn2-mr9vl to exist\nMachine o0z7z4b2j5v2y3n-np-delete-main-bbsn2-ms6dr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-sd447nrw2b6l--managed/providers/Microsoft.Compute/virtualMachines/o0z7z4b2j5v2y3n-np-delete-main-bbsn2-ms6dr to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j0059904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7256f4ec-b394-462d-a65c-c0c44cf424ed
correlation ID: fc291250-1808-46ce-ae93-ea402484d914
client request ID: 0188fcc0-06aa-4fd8-81c3-b6ebc6f67822
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x1261114a4180>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-sd447nrw2b6l" to finish creating: GET https://rp.j0059904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7256f4ec-b394-462d-a65c-c0c44cf424ed
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7256f4ec-b394-462d-a65c-c0c44cf424ed",
"name": "7256f4ec-b394-462d-a65c-c0c44cf424ed",
"status": "Failed",
"startTime": "2026-10-13T16:58:45.675244209Z",
"endTime": "2026-10-13T17:27:48.586418683Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine o0z7z4b2j5v2y3n-np-delete-main-bbsn2-mr9vl: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-sd447nrw2b6l--managed/providers/Microsoft.Compute/virtualMachines/o0z7z4b2j5v2y3n-np-delete-main-bbsn2-mr9vl to exist\nMachine o0z7z4b2j5v2y3n-np-delete-main-bbsn2-ms6dr: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-sd447nrw2b6l--managed/providers/Microsoft.Compute/virtualMachines/o0z7z4b2j5v2y3n-np-delete-main-bbsn2-ms6dr to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine o0z7z4b2j5v2y3n-np-delete-main-bbsn2-mr9vl: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-sd447nrw2b6l--managed/providers/Microsoft.Compute/virtualMachines/o0z7z4b2j5v2y3n-np-delete-main-bbsn2-mr9vl to exist\nMachine o0z7z4b2j5v2y3n-np-delete-main-bbsn2-ms6dr: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-sd447nrw2b6l--managed/providers/Microsoft.Compute/virtualMachines/o0z7z4b2j5v2y3n-np-delete-main-bbsn2-ms6dr to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j0059904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7256f4ec-b394-462d-a65c-c0c44cf424ed
correlation ID: fc291250-1808-46ce-ae93-ea402484d914
client request ID: 0188fcc0-06aa-4fd8-81c3-b6ebc6f67822ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
<*fmt.wrapError | 0x37d94a08f940>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-j6jbclq8czt4" to finish creating: GET https://rp.j0059904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f09b4f7f-dd14-4709-a93e-1fc185bebf4a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f09b4f7f-dd14-4709-a93e-1fc185bebf4a",
"name": "f09b4f7f-dd14-4709-a93e-1fc185bebf4a",
"status": "Failed",
"startTime": "2026-10-13T17:16:34.496076904Z",
"endTime": "2026-10-13T17:45:42.970608528Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r8o8t5e4i0j8y1g-np-1-mc4v9-67mf2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-j6jbclq8czt4--managed/providers/Microsoft.Compute/virtualMachines/r8o8t5e4i0j8y1g-np-1-mc4v9-67mf2 to exist\nMachine r8o8t5e4i0j8y1g-np-1-mc4v9-kxdpm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-j6jbclq8czt4--managed/providers/Microsoft.Compute/virtualMachines/r8o8t5e4i0j8y1g-np-1-mc4v9-kxdpm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r8o8t5e4i0j8y1g-np-1-mc4v9-67mf2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-j6jbclq8czt4--managed/providers/Microsoft.Compute/virtualMachines/r8o8t5e4i0j8y1g-np-1-mc4v9-67mf2 to exist\nMachine r8o8t5e4i0j8y1g-np-1-mc4v9-kxdpm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-j6jbclq8czt4--managed/providers/Microsoft.Compute/virtualMachines/r8o8t5e4i0j8y1g-np-1-mc4v9-kxdpm to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j0059904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f09b4f7f-dd14-4709-a93e-1fc185bebf4a
correlation ID: fa9a1781-c6d1-4176-bdee-881a0f4c4796
client request ID: 94e1b016-c599-42c0-a81c-eb85c23ce070
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:159]: failed to create node pool np-1
Unexpected error:
<*fmt.wrapError | 0x37d94a08f940>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "negative-tests-cluster" in resourcegroup="rg-negative-tests-j6jbclq8czt4" to finish creating: GET https://rp.j0059904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f09b4f7f-dd14-4709-a93e-1fc185bebf4a
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f09b4f7f-dd14-4709-a93e-1fc185bebf4a",
"name": "f09b4f7f-dd14-4709-a93e-1fc185bebf4a",
"status": "Failed",
"startTime": "2026-10-13T17:16:34.496076904Z",
"endTime": "2026-10-13T17:45:42.970608528Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine r8o8t5e4i0j8y1g-np-1-mc4v9-67mf2: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-j6jbclq8czt4--managed/providers/Microsoft.Compute/virtualMachines/r8o8t5e4i0j8y1g-np-1-mc4v9-67mf2 to exist\nMachine r8o8t5e4i0j8y1g-np-1-mc4v9-kxdpm: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-j6jbclq8czt4--managed/providers/Microsoft.Compute/virtualMachines/r8o8t5e4i0j8y1g-np-1-mc4v9-kxdpm to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine r8o8t5e4i0j8y1g-np-1-mc4v9-67mf2: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-j6jbclq8czt4--managed/providers/Microsoft.Compute/virtualMachines/r8o8t5e4i0j8y1g-np-1-mc4v9-67mf2 to exist\nMachine r8o8t5e4i0j8y1g-np-1-mc4v9-kxdpm: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-j6jbclq8czt4--managed/providers/Microsoft.Compute/virtualMachines/r8o8t5e4i0j8y1g-np-1-mc4v9-kxdpm to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j0059904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f09b4f7f-dd14-4709-a93e-1fc185bebf4a
correlation ID: fa9a1781-c6d1-4176-bdee-881a0f4c4796
client request ID: 94e1b016-c599-42c0-a81c-eb85c23ce070
occurred |
|
| 2h 20m 3s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
centralus | alert, e2e | failed | #7489 (merged) | 2 | e2e (1)ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x31b87eaf00d8>:
resource group "node-limit-test-2ch85v6j6t94": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-2ch85v6j6t94" to finish deleting: GET https://rp.j9475584.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2de5bbd0-8d23-4d85-8e58-9564e3649e84
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2de5bbd0-8d23-4d85-8e58-9564e3649e84",
"name": "2de5bbd0-8d23-4d85-8e58-9564e3649e84",
"status": "Failed",
"startTime": "2026-10-13T16:37:18.781083053Z",
"endTime": "2026-10-13T17:01:24.31591322Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0s3j58fj1f3fm4vsgrm1lrp3it3l1 still exists (deletion dispatched at 2026-10-13T16:37:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9475584.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2de5bbd0-8d23-4d85-8e58-9564e3649e84
correlation ID: 879b6302-da33-4ea2-941e-e29845fe6259
client request ID: f5a45ff4-173b-4f67-a8d6-7bf91987cdf6
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x31b87eaf00d8>:
resource group "node-limit-test-2ch85v6j6t94": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-2ch85v6j6t94" to finish deleting: GET https://rp.j9475584.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2de5bbd0-8d23-4d85-8e58-9564e3649e84
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2de5bbd0-8d23-4d85-8e58-9564e3649e84",
"name": "2de5bbd0-8d23-4d85-8e58-9564e3649e84",
"status": "Failed",
"startTime": "2026-10-13T16:37:18.781083053Z",
"endTime": "2026-10-13T17:01:24.31591322Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0s3j58fj1f3fm4vsgrm1lrp3it3l1 still exists (deletion dispatched at 2026-10-13T16:37:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9475584.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2de5bbd0-8d23-4d85-8e58-9564e3649e84
correlation ID: 879b6302-da33-4ea2-941e-e29845fe6259
client request ID: f5a45ff4-173b-4f67-a8d6-7bf91987cdf6
occurredalert (1)alert [svc] LeaderElectionLeaseStale fired Show raw failurealert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T17:22:03Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j9475584-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.62:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9475584/providers/microsoft.monitor/accounts/services-j9475584", namespace="klusterlet-2te0sh8st5vlk5p86ivnmeq75dqqc3c7", pod="arohcp-monitor-kube-state-metrics-7988dc648c-gptdj", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2te0sh8st5vlk5p86ivnmeq75dqqc3c7 on cluster ci01-j9475584-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. |
|
| 2h 7m 26s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | e2e | failed | #7503 (merged) | 2 | e2e (2)ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-bpvl86" for cluster "cluster-hs-pre-bpvl86"
Unexpected error:
<*fmt.wrapError | 0x1d5e48e6be00>:
failed to create NodePool np-bpvl86: failed waiting for nodepool="np-bpvl86" for cluster "cluster-hs-pre-bpvl86" in resourcegroup="rg-hs-pre--94dxdngtsz44" to finish creating: GET https://rp.j3717760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83a4ff43-825f-4641-9a90-75df22dbb575
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83a4ff43-825f-4641-9a90-75df22dbb575",
"name": "83a4ff43-825f-4641-9a90-75df22dbb575",
"status": "Failed",
"startTime": "2026-10-13T16:46:39.5702476Z",
"endTime": "2026-10-13T17:15:41.984565887Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w2q8t7x4u8t3v0q-np-bpvl86-wnkxp-4lslk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--94dxdngtsz44-bpvl86--managed/providers/Microsoft.Compute/virtualMachines/w2q8t7x4u8t3v0q-np-bpvl86-wnkxp-4lslk to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w2q8t7x4u8t3v0q-np-bpvl86-wnkxp-4lslk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--94dxdngtsz44-bpvl86--managed/providers/Microsoft.Compute/virtualMachines/w2q8t7x4u8t3v0q-np-bpvl86-wnkxp-4lslk to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3717760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83a4ff43-825f-4641-9a90-75df22dbb575
correlation ID: 4c25f049-32a9-4598-b0f0-3e14c758ee5b
client request ID: c16c3d79-0122-4655-b7de-468b0afbb13d
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:112]: failed to create node pool "np-bpvl86" for cluster "cluster-hs-pre-bpvl86"
Unexpected error:
<*fmt.wrapError | 0x1d5e48e6be00>:
failed to create NodePool np-bpvl86: failed waiting for nodepool="np-bpvl86" for cluster "cluster-hs-pre-bpvl86" in resourcegroup="rg-hs-pre--94dxdngtsz44" to finish creating: GET https://rp.j3717760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83a4ff43-825f-4641-9a90-75df22dbb575
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83a4ff43-825f-4641-9a90-75df22dbb575",
"name": "83a4ff43-825f-4641-9a90-75df22dbb575",
"status": "Failed",
"startTime": "2026-10-13T16:46:39.5702476Z",
"endTime": "2026-10-13T17:15:41.984565887Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine w2q8t7x4u8t3v0q-np-bpvl86-wnkxp-4lslk: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--94dxdngtsz44-bpvl86--managed/providers/Microsoft.Compute/virtualMachines/w2q8t7x4u8t3v0q-np-bpvl86-wnkxp-4lslk to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine w2q8t7x4u8t3v0q-np-bpvl86-wnkxp-4lslk: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-hs-pre--94dxdngtsz44-bpvl86--managed/providers/Microsoft.Compute/virtualMachines/w2q8t7x4u8t3v0q-np-bpvl86-wnkxp-4lslk to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3717760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/83a4ff43-825f-4641-9a90-75df22dbb575
correlation ID: 4c25f049-32a9-4598-b0f0-3e14c758ee5b
client request ID: c16c3d79-0122-4655-b7de-468b0afbb13d
occurredfailed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x14c5067047e0>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-wgpx2l8rb8pt" to finish creating: GET https://rp.j3717760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bfdbd7d1-4678-4b27-805d-06169447d222
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bfdbd7d1-4678-4b27-805d-06169447d222",
"name": "bfdbd7d1-4678-4b27-805d-06169447d222",
"status": "Failed",
"startTime": "2026-10-13T16:50:30.459831291Z",
"endTime": "2026-10-13T17:19:31.991486899Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f9x2z8c7e8j4t1k-np-delete-main-nf26v-k995z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wgpx2l8rb8pt--managed/providers/Microsoft.Compute/virtualMachines/f9x2z8c7e8j4t1k-np-delete-main-nf26v-k995z to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f9x2z8c7e8j4t1k-np-delete-main-nf26v-k995z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wgpx2l8rb8pt--managed/providers/Microsoft.Compute/virtualMachines/f9x2z8c7e8j4t1k-np-delete-main-nf26v-k995z to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3717760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bfdbd7d1-4678-4b27-805d-06169447d222
correlation ID: 7f043b84-b98d-4c03-bf38-f6fdade9443c
client request ID: 05384f3e-2de3-496b-af3e-0834a7a0f0d2
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
<*fmt.wrapError | 0x14c5067047e0>:
failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-wgpx2l8rb8pt" to finish creating: GET https://rp.j3717760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bfdbd7d1-4678-4b27-805d-06169447d222
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bfdbd7d1-4678-4b27-805d-06169447d222",
"name": "bfdbd7d1-4678-4b27-805d-06169447d222",
"status": "Failed",
"startTime": "2026-10-13T16:50:30.459831291Z",
"endTime": "2026-10-13T17:19:31.991486899Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine f9x2z8c7e8j4t1k-np-delete-main-nf26v-k995z: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wgpx2l8rb8pt--managed/providers/Microsoft.Compute/virtualMachines/f9x2z8c7e8j4t1k-np-delete-main-nf26v-k995z to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine f9x2z8c7e8j4t1k-np-delete-main-nf26v-k995z: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-wgpx2l8rb8pt--managed/providers/Microsoft.Compute/virtualMachines/f9x2z8c7e8j4t1k-np-delete-main-nf26v-k995z to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j3717760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bfdbd7d1-4678-4b27-805d-06169447d222
correlation ID: 7f043b84-b98d-4c03-bf38-f6fdade9443c
client request ID: 05384f3e-2de3-496b-af3e-0834a7a0f0d2 |
|
| 2h 13m 17s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
westus3 | alert | failed | #7501 (merged) | 1 | alert (1)alert [svc] BackendControllerQueueDepthHigh fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T16:50:39Z Ended: 2026-10-13T16:56:39Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j1238272-svc", component="backend", name="clusterresources", region="westus3", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. |
|
| 2h 20m 23s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | westus3 | n/a | passed | #7497 (open) | 0 | n/a | |
| 2h 28m 13s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | n/a | passed | #6092 (merged) | 0 | n/a | |
| 2h 26m 48s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | westus3 | alert | failed | #7499 (open) | 2 | alert (2)alert [svc] BackendControllerQueueDepthHigh fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T16:44:34Z Ended: 2026-10-13T16:55:36Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7907968-svc", component="backend", name="clusterresources", region="westus3", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert [svc] LeaderElectionLeaseStale fired Show raw failurealert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T17:32:44Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j7907968-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.161:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7907968/providers/microsoft.monitor/accounts/services-j7907968", namespace="klusterlet-2te0u0kbkndd5ajbfrtkgsggtl7jvgde", pod="arohcp-monitor-kube-state-metrics-7988dc648c-df6h4", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2te0u0kbkndd5ajbfrtkgsggtl7jvgde on cluster ci01-j7907968-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. Firing 2: State: Fired (not resolved) Started: 2026-10-13T17:36:44Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j7907968-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.161:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7907968/providers/microsoft.monitor/accounts/services-j7907968", namespace="klusterlet-2te0tfu3d0rbgc2bl5u7fshuhidhum5l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-df6h4", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2te0tfu3d0rbgc2bl5u7fshuhidhum5l on cluster ci01-j7907968-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. |
|
| 2h 20m 3s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | alert, e2e | failed | #7496 (merged) | 20 | e2e (19)ERROR CODE: DeploymentFailed; detail code InternalServerError; detail message An error occurred. Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:112]: failed to create customer resources for cluster "cluster-candidate-4-23-x7n4rm"
Unexpected error:
<*fmt.wrapError | 0x1df325fdde20>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-candidate-4-23-x7n4rm-mgfdjq" in resourcegroup="rg-candidate-4-23-25wqkbxjvqpm" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-23-25wqkbxjvqpm/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-23-x7n4rm-mgfdjq/operationStatuses/08584096986588873020
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"InternalServerError\",\r\n \"message\": \"An error occurred.\",\r\n \"details\": []\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-23-25wqkbxjvqpm/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-23-x7n4rm-mgfdjq/operationStatuses/08584096986588873020
correlation ID: 66a77b00-0c74-46a1-bab1-d73fd9784240
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:112]: failed to create customer resources for cluster "cluster-candidate-4-23-x7n4rm"
Unexpected error:
<*fmt.wrapError | 0x1df325fdde20>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-candidate-4-23-x7n4rm-mgfdjq" in resourcegroup="rg-candidate-4-23-25wqkbxjvqpm" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-23-25wqkbxjvqpm/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-23-x7n4rm-mgfdjq/operationStatuses/08584096986588873020
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"InternalServerError\",\r\n \"message\": \"An error occurred.\",\r\n \"details\": []\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-candidate-4-23-25wqkbxjvqpm/providers/Microsoft.Resources/deployments/customer-infra-cluster-candidate-4-23-x7n4rm-mgfdjq/operationStatuses/08584096986588873020
correlation ID: 66a77b00-0c74-46a1-bab1-d73fd9784240
client request ID: unavailable
occurredERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x309531b0198>:
resource group "ea-list-wtljvtmmhcsk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-list8h4wvd" in resourcegroup="ea-list-wtljvtmmhcsk" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2f74cf89-c5af-4bbc-9410-e14470aca714
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2f74cf89-c5af-4bbc-9410-e14470aca714",
"name": "2f74cf89-c5af-4bbc-9410-e14470aca714",
"status": "Failed",
"startTime": "2026-10-13T16:38:18.149454561Z",
"endTime": "2026-10-13T17:02:21.62230875Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0o6snbp7v915nacod7cib95rmn96j still exists (deletion dispatched at 2026-10-13T16:38:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2f74cf89-c5af-4bbc-9410-e14470aca714
correlation ID: 8d339200-f0e0-47b8-90d7-53a61a990c91
client request ID: 742fa150-0903-4156-b9ea-5d97b505f86c
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x309531b0198>:
resource group "ea-list-wtljvtmmhcsk": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-list8h4wvd" in resourcegroup="ea-list-wtljvtmmhcsk" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2f74cf89-c5af-4bbc-9410-e14470aca714
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2f74cf89-c5af-4bbc-9410-e14470aca714",
"name": "2f74cf89-c5af-4bbc-9410-e14470aca714",
"status": "Failed",
"startTime": "2026-10-13T16:38:18.149454561Z",
"endTime": "2026-10-13T17:02:21.62230875Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0o6snbp7v915nacod7cib95rmn96j still exists (deletion dispatched at 2026-10-13T16:38:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2f74cf89-c5af-4bbc-9410-e14470aca714
correlation ID: 8d339200-f0e0-47b8-90d7-53a61a990c91
client request ID: 742fa150-0903-4156-b9ea-5d97b505f86c
occurredERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x86c6d66c288>:
resource group "ephemeral-osdisk-zk6g6jlt9pcm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-zk6g6jlt9pcm" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e42b64f2-dbc7-46c0-ade3-38701934a844
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e42b64f2-dbc7-46c0-ade3-38701934a844",
"name": "e42b64f2-dbc7-46c0-ade3-38701934a844",
"status": "Failed",
"startTime": "2026-10-13T16:37:24.341394968Z",
"endTime": "2026-10-13T17:01:31.660742691Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0o0fo95vc6ql30rb17lllmie9v71c still exists (deletion dispatched at 2026-10-13T16:37:25Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e42b64f2-dbc7-46c0-ade3-38701934a844
correlation ID: c01082db-7899-4024-8f97-9610dd939a4e
client request ID: 1f3a6602-152b-48ad-934d-22af532614cf
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x86c6d66c288>:
resource group "ephemeral-osdisk-zk6g6jlt9pcm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-zk6g6jlt9pcm" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e42b64f2-dbc7-46c0-ade3-38701934a844
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e42b64f2-dbc7-46c0-ade3-38701934a844",
"name": "e42b64f2-dbc7-46c0-ade3-38701934a844",
"status": "Failed",
"startTime": "2026-10-13T16:37:24.341394968Z",
"endTime": "2026-10-13T17:01:31.660742691Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0o0fo95vc6ql30rb17lllmie9v71c still exists (deletion dispatched at 2026-10-13T16:37:25Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e42b64f2-dbc7-46c0-ade3-38701934a844
correlation ID: c01082db-7899-4024-8f97-9610dd939a4e
client request ID: 1f3a6602-152b-48ad-934d-22af532614cf
occurredDeleteHCPCluster timed out after <minutes> minutes; context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x36cea0d80120>:
resource group "rg-cluster-back-version-4285twt9lhf2": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-4285twt9lhf2", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-4285twt9lhf2" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-4285twt9lhf2" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x36ce9f324380>{
msg: "resource group \"rg-cluster-back-version-4285twt9lhf2\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded",
err: <*fmt.wrapError | 0x36ce9f324360>{
msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded",
err: <*fmt.wrapErrors | 0x36cea0a86750>{
msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded",
errs: [
<*fmt.wrapErrors | 0x36ce9f315470>{
msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x36ce9f860ef0>{
s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2",
},
<context.deadlineExceededError>{},
],
},
<*fmt.wrapErrors | 0x36ce9f315470>{
msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x36ce9f860ef0>{
s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2",
},
<context.deadlineExceededError>{},
],
},
],
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x36cea0d80120>:
resource group "rg-cluster-back-version-4285twt9lhf2": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup="rg-cluster-back-version-4285twt9lhf2", caused by: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-4285twt9lhf2" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded, error: failed waiting for hcpCluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-4285twt9lhf2" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x36ce9f324380>{
msg: "resource group \"rg-cluster-back-version-4285twt9lhf2\": failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded",
err: <*fmt.wrapError | 0x36ce9f324360>{
msg: "failed to cleanup resource group: failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded",
err: <*fmt.wrapErrors | 0x36cea0a86750>{
msg: "failed deleting hcp clusters in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\", caused by: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded, error: failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded",
errs: [
<*fmt.wrapErrors | 0x36ce9f315470>{
msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x36ce9f860ef0>{
s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2",
},
<context.deadlineExceededError>{},
],
},
<*fmt.wrapErrors | 0x36ce9f315470>{
msg: "failed waiting for hcpCluster=\"cluster-ver-4-20\" in resourcegroup=\"rg-cluster-back-version-4285twt9lhf2\" to finish deleting, caused by: timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2, error: context deadline exceeded",
errs: [
<*errors.errorString | 0x36ce9f860ef0>{
s: "timeout '25.000000' minutes exceeded during DeleteHCPCluster for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-4285twt9lhf2",
},
<context.deadlineExceededError>{},
],
},
],
},
},
},
],
}
occurredERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x2e91a04e0060>:
resource group "patch-tags-4btcwtspkn4z": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-4btcwtspkn4z" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/487410f3-b72a-4fc3-9193-c1517e85efb3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/487410f3-b72a-4fc3-9193-c1517e85efb3",
"name": "487410f3-b72a-4fc3-9193-c1517e85efb3",
"status": "Failed",
"startTime": "2026-10-13T16:37:24.68179111Z",
"endTime": "2026-10-13T17:01:31.676853999Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0nltmn5utnsjpt0ljsht3622t7msm still exists (deletion dispatched at 2026-10-13T16:37:25Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/487410f3-b72a-4fc3-9193-c1517e85efb3
correlation ID: 39a9fc6d-ba22-4803-9010-4b78f2a6492a
client request ID: 8d74e002-092e-41b2-a1b9-73b32b63b7fa
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x2e91a04e0060>:
resource group "patch-tags-4btcwtspkn4z": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-4btcwtspkn4z" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/487410f3-b72a-4fc3-9193-c1517e85efb3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/487410f3-b72a-4fc3-9193-c1517e85efb3",
"name": "487410f3-b72a-4fc3-9193-c1517e85efb3",
"status": "Failed",
"startTime": "2026-10-13T16:37:24.68179111Z",
"endTime": "2026-10-13T17:01:31.676853999Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0nltmn5utnsjpt0ljsht3622t7msm still exists (deletion dispatched at 2026-10-13T16:37:25Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/487410f3-b72a-4fc3-9193-c1517e85efb3
correlation ID: 39a9fc6d-ba22-4803-9010-4b78f2a6492a
client request ID: 8d74e002-092e-41b2-a1b9-73b32b63b7fa
occurredERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x3646ea5bc090>:
resource group "node-limit-test-8kxdfg5cbrwp": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-8kxdfg5cbrwp" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5a7b5709-9d05-492c-bf2b-e88948da705b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5a7b5709-9d05-492c-bf2b-e88948da705b",
"name": "5a7b5709-9d05-492c-bf2b-e88948da705b",
"status": "Failed",
"startTime": "2026-10-13T16:38:27.169390438Z",
"endTime": "2026-10-13T17:02:31.60705305Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0nvvfqaanfc2udupbrnjbsmglpsr1 still exists (deletion dispatched at 2026-10-13T16:38:28Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5a7b5709-9d05-492c-bf2b-e88948da705b
correlation ID: 8e472778-6c92-4350-b17d-60c0f029163b
client request ID: abc90e83-96e9-4095-a014-84d0c0f96649
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x3646ea5bc090>:
resource group "node-limit-test-8kxdfg5cbrwp": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="node-limit-cluster" in resourcegroup="node-limit-test-8kxdfg5cbrwp" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5a7b5709-9d05-492c-bf2b-e88948da705b
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5a7b5709-9d05-492c-bf2b-e88948da705b",
"name": "5a7b5709-9d05-492c-bf2b-e88948da705b",
"status": "Failed",
"startTime": "2026-10-13T16:38:27.169390438Z",
"endTime": "2026-10-13T17:02:31.60705305Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0nvvfqaanfc2udupbrnjbsmglpsr1 still exists (deletion dispatched at 2026-10-13T16:38:28Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5a7b5709-9d05-492c-bf2b-e88948da705b
correlation ID: 8e472778-6c92-4350-b17d-60c0f029163b
client request ID: abc90e83-96e9-4095-a014-84d0c0f96649
occurredERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x3c5bff2321b0>:
resource group "rg-candidate-4-21-ckfqmkfrhb7s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-21-x2ltjj" in resourcegroup="rg-candidate-4-21-ckfqmkfrhb7s" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/35900dd3-f76a-4ad6-8666-ea0b4b1885cd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/35900dd3-f76a-4ad6-8666-ea0b4b1885cd",
"name": "35900dd3-f76a-4ad6-8666-ea0b4b1885cd",
"status": "Failed",
"startTime": "2026-10-13T16:36:24.431313337Z",
"endTime": "2026-10-13T17:00:31.622971609Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0o6eoe74gqn4aq2k2plogghjg4a3t still exists (deletion dispatched at 2026-10-13T16:36:24Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/35900dd3-f76a-4ad6-8666-ea0b4b1885cd
correlation ID: 8a956502-ed8a-4638-9158-d916f64e4232
client request ID: 5f464c51-9b3d-4872-ac2e-f49527555192
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x3c5bff2321b0>:
resource group "rg-candidate-4-21-ckfqmkfrhb7s": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-21-x2ltjj" in resourcegroup="rg-candidate-4-21-ckfqmkfrhb7s" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/35900dd3-f76a-4ad6-8666-ea0b4b1885cd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/35900dd3-f76a-4ad6-8666-ea0b4b1885cd",
"name": "35900dd3-f76a-4ad6-8666-ea0b4b1885cd",
"status": "Failed",
"startTime": "2026-10-13T16:36:24.431313337Z",
"endTime": "2026-10-13T17:00:31.622971609Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0o6eoe74gqn4aq2k2plogghjg4a3t still exists (deletion dispatched at 2026-10-13T16:36:24Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/35900dd3-f76a-4ad6-8666-ea0b4b1885cd
correlation ID: 8a956502-ed8a-4638-9158-d916f64e4232
client request ID: 5f464c51-9b3d-4872-ac2e-f49527555192
occurredERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x223786a70108>:
resource group "rg-candidate-5-0-l4hscszf2rxz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-lvqr8d" in resourcegroup="rg-candidate-5-0-l4hscszf2rxz" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b44ea302-7f97-4969-a60c-eec855852cac
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b44ea302-7f97-4969-a60c-eec855852cac",
"name": "b44ea302-7f97-4969-a60c-eec855852cac",
"status": "Failed",
"startTime": "2026-10-13T16:40:42.747363753Z",
"endTime": "2026-10-13T17:04:51.623105937Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0o054eslakn49l68pthqigeds4ule still exists (deletion dispatched at 2026-10-13T16:40:43Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b44ea302-7f97-4969-a60c-eec855852cac
correlation ID: 4bfc0959-e629-40fb-9245-01ec4651bd6b
client request ID: a781f5bf-303e-4280-ba8a-44cc833b6ce4
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x223786a70108>:
resource group "rg-candidate-5-0-l4hscszf2rxz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-5-0-lvqr8d" in resourcegroup="rg-candidate-5-0-l4hscszf2rxz" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b44ea302-7f97-4969-a60c-eec855852cac
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b44ea302-7f97-4969-a60c-eec855852cac",
"name": "b44ea302-7f97-4969-a60c-eec855852cac",
"status": "Failed",
"startTime": "2026-10-13T16:40:42.747363753Z",
"endTime": "2026-10-13T17:04:51.623105937Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0o054eslakn49l68pthqigeds4ule still exists (deletion dispatched at 2026-10-13T16:40:43Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b44ea302-7f97-4969-a60c-eec855852cac
correlation ID: 4bfc0959-e629-40fb-9245-01ec4651bd6b
client request ID: a781f5bf-303e-4280-ba8a-44cc833b6ce4
occurredERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x33c5e58a20c0>:
resource group "private-keyvault-tbl82js9plw4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-tbl82js9plw4" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e616d6b4-805f-435f-aa67-d2ddddd5c316
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e616d6b4-805f-435f-aa67-d2ddddd5c316",
"name": "e616d6b4-805f-435f-aa67-d2ddddd5c316",
"status": "Failed",
"startTime": "2026-10-13T16:35:08.583263738Z",
"endTime": "2026-10-13T16:59:11.97611737Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0op2sf7jll6tp2tplqbte694tvbr6 still exists (deletion dispatched at 2026-10-13T16:35:09Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e616d6b4-805f-435f-aa67-d2ddddd5c316
correlation ID: 41573f31-0b63-4f6a-92a1-29c5b34c8e25
client request ID: 94ebdfbe-0afd-4bb5-b0fc-3201e548fa3c
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x33c5e58a20c0>:
resource group "private-keyvault-tbl82js9plw4": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="private-kv-cluster" in resourcegroup="private-keyvault-tbl82js9plw4" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e616d6b4-805f-435f-aa67-d2ddddd5c316
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e616d6b4-805f-435f-aa67-d2ddddd5c316",
"name": "e616d6b4-805f-435f-aa67-d2ddddd5c316",
"status": "Failed",
"startTime": "2026-10-13T16:35:08.583263738Z",
"endTime": "2026-10-13T16:59:11.97611737Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0op2sf7jll6tp2tplqbte694tvbr6 still exists (deletion dispatched at 2026-10-13T16:35:09Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e616d6b4-805f-435f-aa67-d2ddddd5c316
correlation ID: 41573f31-0b63-4f6a-92a1-29c5b34c8e25
client request ID: 94ebdfbe-0afd-4bb5-b0fc-3201e548fa3c
occurredERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.41
Unexpected error:
<*fmt.wrapError | 0x343c841917c0>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-ld8wft" in resourcegroup="rg-np-noedge-ld8wft-lrx6cgcqbn7h" to finish creating: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6b0653ae-307f-47d1-8798-9271b8c88bd6
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceNotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceNotFound",
"message": "The resource 'locations/hcpOperationStatuses/6b0653ae-307f-47d1-8798-9271b8c88bd6' under resource group '' was not found.",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6b0653ae-307f-47d1-8798-9271b8c88bd6"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6b0653ae-307f-47d1-8798-9271b8c88bd6
correlation ID: ffecaa7d-bd14-4598-8eba-697dbec64dbe
client request ID: 41765bff-079d-45be-ae54-98a967650838
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:321]: failed to create nodepool npnoedge-4-20 at version 4.20.41
Unexpected error:
<*fmt.wrapError | 0x343c841917c0>:
failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-ld8wft" in resourcegroup="rg-np-noedge-ld8wft-lrx6cgcqbn7h" to finish creating: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6b0653ae-307f-47d1-8798-9271b8c88bd6
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceNotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceNotFound",
"message": "The resource 'locations/hcpOperationStatuses/6b0653ae-307f-47d1-8798-9271b8c88bd6' under resource group '' was not found.",
"target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6b0653ae-307f-47d1-8798-9271b8c88bd6"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6b0653ae-307f-47d1-8798-9271b8c88bd6
correlation ID: ffecaa7d-bd14-4598-8eba-697dbec64dbe
client request ID: 41765bff-079d-45be-ae54-98a967650838
occurredERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x1bf360d7c078>:
resource group "delayed-rbac-86jnlmj4hp5c": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-86jnlmj4hp5c" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b559cd85-38ce-4c5c-97e2-77f3a2b7b2b4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b559cd85-38ce-4c5c-97e2-77f3a2b7b2b4",
"name": "b559cd85-38ce-4c5c-97e2-77f3a2b7b2b4",
"status": "Failed",
"startTime": "2026-10-13T16:32:32.577215674Z",
"endTime": "2026-10-13T16:56:41.960331085Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0nogn7v488ab5md5u7qq776j5f8gl still exists (deletion dispatched at 2026-10-13T16:32:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b559cd85-38ce-4c5c-97e2-77f3a2b7b2b4
correlation ID: 260d0473-ba98-4907-80ca-4905c6dbb05b
client request ID: dca2c435-4751-4ccd-b2b2-4c228a5df5fa
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x1bf360d7c078>:
resource group "delayed-rbac-86jnlmj4hp5c": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-86jnlmj4hp5c" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b559cd85-38ce-4c5c-97e2-77f3a2b7b2b4
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b559cd85-38ce-4c5c-97e2-77f3a2b7b2b4",
"name": "b559cd85-38ce-4c5c-97e2-77f3a2b7b2b4",
"status": "Failed",
"startTime": "2026-10-13T16:32:32.577215674Z",
"endTime": "2026-10-13T16:56:41.960331085Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0nogn7v488ab5md5u7qq776j5f8gl still exists (deletion dispatched at 2026-10-13T16:32:32Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b559cd85-38ce-4c5c-97e2-77f3a2b7b2b4
correlation ID: 260d0473-ba98-4907-80ca-4905c6dbb05b
client request ID: dca2c435-4751-4ccd-b2b2-4c228a5df5fa
occurredERROR CODE: DeploymentFailed; detail code InternalServerError; detail message An error occurred. Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:123]: failed to create customer resources for y-stream upgrade cluster "cp-ystream-upgrade-4-21-gpscnh"
Unexpected error:
<*fmt.wrapError | 0x19351c9c25a0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cp-ystream-upgrade-4-21-gpscnh-6d9mtn" in resourcegroup="rg-cp-ystream-upgrade-4-21-q2fxp7w8nsb4" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cp-ystream-upgrade-4-21-q2fxp7w8nsb4/providers/Microsoft.Resources/deployments/customer-infra-cp-ystream-upgrade-4-21-gpscnh-6d9mtn/operationStatuses/08584096986507070624
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"InternalServerError\",\r\n \"message\": \"An error occurred.\",\r\n \"details\": []\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cp-ystream-upgrade-4-21-q2fxp7w8nsb4/providers/Microsoft.Resources/deployments/customer-infra-cp-ystream-upgrade-4-21-gpscnh-6d9mtn/operationStatuses/08584096986507070624
correlation ID: 73cd0e3a-2347-4fb2-a937-dc27e30c4de2
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:123]: failed to create customer resources for y-stream upgrade cluster "cp-ystream-upgrade-4-21-gpscnh"
Unexpected error:
<*fmt.wrapError | 0x19351c9c25a0>:
failed to create customer-infra: failed waiting for deployment "customer-infra-cp-ystream-upgrade-4-21-gpscnh-6d9mtn" in resourcegroup="rg-cp-ystream-upgrade-4-21-q2fxp7w8nsb4" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cp-ystream-upgrade-4-21-q2fxp7w8nsb4/providers/Microsoft.Resources/deployments/customer-infra-cp-ystream-upgrade-4-21-gpscnh-6d9mtn/operationStatuses/08584096986507070624
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"InternalServerError\",\r\n \"message\": \"An error occurred.\",\r\n \"details\": []\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cp-ystream-upgrade-4-21-q2fxp7w8nsb4/providers/Microsoft.Resources/deployments/customer-infra-cp-ystream-upgrade-4-21-gpscnh-6d9mtn/operationStatuses/08584096986507070624
correlation ID: 73cd0e3a-2347-4fb2-a937-dc27e30c4de2
client request ID: unavailable
occurredERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0xe966a774240>:
resource group "arm64-vm-cluster-6rwgx9dzrn7g": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-6rwgx9dzrn7g" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2b094443-c590-42b9-980a-362abc782d9e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2b094443-c590-42b9-980a-362abc782d9e",
"name": "2b094443-c590-42b9-980a-362abc782d9e",
"status": "Failed",
"startTime": "2026-10-13T16:34:49.54918973Z",
"endTime": "2026-10-13T16:58:51.683203613Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0nl800tudbvnpa73iohd0jebru9cn still exists (deletion dispatched at 2026-10-13T16:34:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2b094443-c590-42b9-980a-362abc782d9e
correlation ID: f88d95b7-3797-479d-9891-e9a44575f535
client request ID: c34e4445-2545-411b-9c29-5bbb95e1b4bb
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0xe966a774240>:
resource group "arm64-vm-cluster-6rwgx9dzrn7g": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-6rwgx9dzrn7g" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2b094443-c590-42b9-980a-362abc782d9e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2b094443-c590-42b9-980a-362abc782d9e",
"name": "2b094443-c590-42b9-980a-362abc782d9e",
"status": "Failed",
"startTime": "2026-10-13T16:34:49.54918973Z",
"endTime": "2026-10-13T16:58:51.683203613Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0nl800tudbvnpa73iohd0jebru9cn still exists (deletion dispatched at 2026-10-13T16:34:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2b094443-c590-42b9-980a-362abc782d9e
correlation ID: f88d95b7-3797-479d-9891-e9a44575f535
client request ID: c34e4445-2545-411b-9c29-5bbb95e1b4bb
occurredERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x2c1d48a96060>:
resource group "disabled-image-registry-7ddljxcvwjcm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-7ddljxcvwjcm" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bf67735b-40e3-4e9f-b12b-b1b7b58e9b9e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bf67735b-40e3-4e9f-b12b-b1b7b58e9b9e",
"name": "bf67735b-40e3-4e9f-b12b-b1b7b58e9b9e",
"status": "Failed",
"startTime": "2026-10-13T16:38:07.060491044Z",
"endTime": "2026-10-13T17:02:11.454465514Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bf67735b-40e3-4e9f-b12b-b1b7b58e9b9e
correlation ID: bc63c490-e9f1-483e-a3e0-dbf6c6ddf72f
client request ID: 93e5d8b4-b42f-4292-ab79-f0ec030e5a84
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x2c1d48a96060>:
resource group "disabled-image-registry-7ddljxcvwjcm": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-7ddljxcvwjcm" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bf67735b-40e3-4e9f-b12b-b1b7b58e9b9e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bf67735b-40e3-4e9f-b12b-b1b7b58e9b9e",
"name": "bf67735b-40e3-4e9f-b12b-b1b7b58e9b9e",
"status": "Failed",
"startTime": "2026-10-13T16:38:07.060491044Z",
"endTime": "2026-10-13T17:02:11.454465514Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bf67735b-40e3-4e9f-b12b-b1b7b58e9b9e
correlation ID: bc63c490-e9f1-483e-a3e0-dbf6c6ddf72f
client request ID: 93e5d8b4-b42f-4292-ab79-f0ec030e5a84
occurredERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x3bf37649c048>:
resource group "customer-rg-kptnhtffptgl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-kptnhtffptgl" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d3fd105c-2a6a-4db9-8a7d-c2c20e4d8b1e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d3fd105c-2a6a-4db9-8a7d-c2c20e4d8b1e",
"name": "d3fd105c-2a6a-4db9-8a7d-c2c20e4d8b1e",
"status": "Failed",
"startTime": "2026-10-13T16:42:39.455530141Z",
"endTime": "2026-10-13T17:06:41.723960509Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0oaptqtlfm45pi2ru69verofh93nj still exists (deletion dispatched at 2026-10-13T16:42:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d3fd105c-2a6a-4db9-8a7d-c2c20e4d8b1e
correlation ID: 57954ad9-2ba3-4943-8c75-f5745f7ae1f7
client request ID: bbb4049f-9eb5-4ee1-9bdd-ccda4efc41c1
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x3bf37649c048>:
resource group "customer-rg-kptnhtffptgl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster" in resourcegroup="customer-rg-kptnhtffptgl" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d3fd105c-2a6a-4db9-8a7d-c2c20e4d8b1e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d3fd105c-2a6a-4db9-8a7d-c2c20e4d8b1e",
"name": "d3fd105c-2a6a-4db9-8a7d-c2c20e4d8b1e",
"status": "Failed",
"startTime": "2026-10-13T16:42:39.455530141Z",
"endTime": "2026-10-13T17:06:41.723960509Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0oaptqtlfm45pi2ru69verofh93nj still exists (deletion dispatched at 2026-10-13T16:42:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d3fd105c-2a6a-4db9-8a7d-c2c20e4d8b1e
correlation ID: 57954ad9-2ba3-4943-8c75-f5745f7ae1f7
client request ID: bbb4049f-9eb5-4ee1-9bdd-ccda4efc41c1
occurredERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x34e38763a138>:
resource group "rg-cordon-test-9l4cgfh8mhfl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-9l4cgfh8mhfl" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ab95bc-886b-4b3d-9a6e-9221165e46bd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ab95bc-886b-4b3d-9a6e-9221165e46bd",
"name": "84ab95bc-886b-4b3d-9a6e-9221165e46bd",
"status": "Failed",
"startTime": "2026-10-13T16:39:24.254400328Z",
"endTime": "2026-10-13T17:03:31.595626382Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0o1agno7465r2vsakhkack392k7ql still exists (deletion dispatched at 2026-10-13T16:39:24Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ab95bc-886b-4b3d-9a6e-9221165e46bd
correlation ID: 9b32bec0-6d45-4533-a8ad-d586fd5de17c
client request ID: 0c2d0a32-aa6c-4c82-a1b0-303746bfb85e
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x34e38763a138>:
resource group "rg-cordon-test-9l4cgfh8mhfl": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-9l4cgfh8mhfl" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ab95bc-886b-4b3d-9a6e-9221165e46bd
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ab95bc-886b-4b3d-9a6e-9221165e46bd",
"name": "84ab95bc-886b-4b3d-9a6e-9221165e46bd",
"status": "Failed",
"startTime": "2026-10-13T16:39:24.254400328Z",
"endTime": "2026-10-13T17:03:31.595626382Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0o1agno7465r2vsakhkack392k7ql still exists (deletion dispatched at 2026-10-13T16:39:24Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/84ab95bc-886b-4b3d-9a6e-9221165e46bd
correlation ID: 9b32bec0-6d45-4533-a8ad-d586fd5de17c
client request ID: 0c2d0a32-aa6c-4c82-a1b0-303746bfb85e
occurredERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x2f1a7aab6150>:
resource group "rg-candidate-4-22-lskfpcc9pllz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-d2g9r5" in resourcegroup="rg-candidate-4-22-lskfpcc9pllz" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f9ed645a-3dcf-42dc-9141-f0f20d1df080
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f9ed645a-3dcf-42dc-9141-f0f20d1df080",
"name": "f9ed645a-3dcf-42dc-9141-f0f20d1df080",
"status": "Failed",
"startTime": "2026-10-13T16:36:46.601277196Z",
"endTime": "2026-10-13T17:00:51.586929721Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0nlj053nsa11ldfoisafgipanrlce still exists (deletion dispatched at 2026-10-13T16:36:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f9ed645a-3dcf-42dc-9141-f0f20d1df080
correlation ID: 39b6fd78-ce95-48e2-8999-56a159d15d97
client request ID: fa481482-679a-4e18-8a64-9dbb028a0bbd
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x2f1a7aab6150>:
resource group "rg-candidate-4-22-lskfpcc9pllz": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-22-d2g9r5" in resourcegroup="rg-candidate-4-22-lskfpcc9pllz" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f9ed645a-3dcf-42dc-9141-f0f20d1df080
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f9ed645a-3dcf-42dc-9141-f0f20d1df080",
"name": "f9ed645a-3dcf-42dc-9141-f0f20d1df080",
"status": "Failed",
"startTime": "2026-10-13T16:36:46.601277196Z",
"endTime": "2026-10-13T17:00:51.586929721Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0nlj053nsa11ldfoisafgipanrlce still exists (deletion dispatched at 2026-10-13T16:36:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f9ed645a-3dcf-42dc-9141-f0f20d1df080
correlation ID: 39b6fd78-ce95-48e2-8999-56a159d15d97
client request ID: fa481482-679a-4e18-8a64-9dbb028a0bbd
occurredERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x1722b271c198>:
resource group "v20260901preview-5x7tmrznp2t6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-5x7tmrznp2t6" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4cd1ca19-0014-4b36-b986-419e0d5b84e1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4cd1ca19-0014-4b36-b986-419e0d5b84e1",
"name": "4cd1ca19-0014-4b36-b986-419e0d5b84e1",
"status": "Failed",
"startTime": "2026-10-13T16:36:15.001844938Z",
"endTime": "2026-10-13T17:00:21.681973204Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0nlh1b0gebk9buu1868a057tlqheh still exists (deletion dispatched at 2026-10-13T16:36:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4cd1ca19-0014-4b36-b986-419e0d5b84e1
correlation ID: 78699c18-7077-4978-876a-af57a1f9b864
client request ID: 3c23b588-05ad-4ee2-9c29-3eca24b5d20c
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x1722b271c198>:
resource group "v20260901preview-5x7tmrznp2t6": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="v20260901" in resourcegroup="v20260901preview-5x7tmrznp2t6" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4cd1ca19-0014-4b36-b986-419e0d5b84e1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4cd1ca19-0014-4b36-b986-419e0d5b84e1",
"name": "4cd1ca19-0014-4b36-b986-419e0d5b84e1",
"status": "Failed",
"startTime": "2026-10-13T16:36:15.001844938Z",
"endTime": "2026-10-13T17:00:21.681973204Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0nlh1b0gebk9buu1868a057tlqheh still exists (deletion dispatched at 2026-10-13T16:36:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4cd1ca19-0014-4b36-b986-419e0d5b84e1
correlation ID: 78699c18-7077-4978-876a-af57a1f9b864
client request ID: 3c23b588-05ad-4ee2-9c29-3eca24b5d20c
occurredERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x39bb325a6558>:
resource group "admin-api-serialconsole-dwh828chkx9c": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-dwh828chkx9c" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0074eb0c-2b24-41cb-a7ff-604c4e76ad09
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0074eb0c-2b24-41cb-a7ff-604c4e76ad09",
"name": "0074eb0c-2b24-41cb-a7ff-604c4e76ad09",
"status": "Failed",
"startTime": "2026-10-13T16:33:20.010489415Z",
"endTime": "2026-10-13T16:57:21.684870881Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0nldrfocqarmt0r34ht6jph1m9d19 still exists (deletion dispatched at 2026-10-13T16:33:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0074eb0c-2b24-41cb-a7ff-604c4e76ad09
correlation ID: d6c08569-5a60-4a0f-bb76-207075ebe9e3
client request ID: f4713fe8-8c37-423d-a5df-6a24a2af5b2d
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:309]: Unexpected error:
<*errors.joinError | 0x39bb325a6558>:
resource group "admin-api-serialconsole-dwh828chkx9c": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-dwh828chkx9c" to finish deleting: GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0074eb0c-2b24-41cb-a7ff-604c4e76ad09
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0074eb0c-2b24-41cb-a7ff-604c4e76ad09",
"name": "0074eb0c-2b24-41cb-a7ff-604c4e76ad09",
"status": "Failed",
"startTime": "2026-10-13T16:33:20.010489415Z",
"endTime": "2026-10-13T16:57:21.684870881Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2te0nldrfocqarmt0r34ht6jph1m9d19 still exists (deletion dispatched at 2026-10-13T16:33:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j6938112.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0074eb0c-2b24-41cb-a7ff-604c4e76ad09
correlation ID: d6c08569-5a60-4a0f-bb76-207075ebe9e3
client request ID: f4713fe8-8c37-423d-a5df-6a24a2af5b2d
occurredalert (1)alert [svc] LeaderElectionLeaseStale fired Show raw failurealert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T17:33:37Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j6938112-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.198:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6938112/providers/microsoft.monitor/accounts/services-j6938112", namespace="klusterlet-2te0oroei8rqcmb2im244arlb9dt9evs", pod="arohcp-monitor-kube-state-metrics-7988dc648c-ztvsb", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2te0oroei8rqcmb2im244arlb9dt9evs on cluster ci01-j6938112-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. |
|
| 4h 30m 11s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | westus3 | n/a | passed | #7498 (open) | 0 | n/a | |
| 15m 53s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | alert | failed | #7501 (merged) | 3 | alert (3)ERROR CODE: ResourceGroupNotFound; provider Microsoft.Insights Show raw failurealert collection incomplete
failed to fetch metric alert rules for /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-usw3: failed to list metric alert rules: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-usw3/providers/Microsoft.Insights/metricAlerts
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceGroupNotFound",
"message": "Resource group 'hcp-underlay-ci01-usw3' could not be found."
}
}
--------------------------------------------------------------------------------ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement Show raw failurealert collection incomplete
failed to fetch hcp alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-usw3/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceGroupNotFound",
"message": "Resource group 'hcp-underlay-ci01-usw3' could not be found."
}
}
--------------------------------------------------------------------------------ERROR CODE: ResourceGroupNotFound; provider Microsoft.AlertsManagement Show raw failurealert collection incomplete
failed to fetch svc alert rules: failed to list prometheus rule groups: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-usw3/providers/Microsoft.AlertsManagement/prometheusRuleGroups
--------------------------------------------------------------------------------
RESPONSE 404: 404 Not Found
ERROR CODE: ResourceGroupNotFound
--------------------------------------------------------------------------------
{
"error": {
"code": "ResourceGroupNotFound",
"message": "Resource group 'hcp-underlay-ci01-usw3' could not be found."
}
}
-------------------------------------------------------------------------------- |
|
| 2h 12m 33s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | westus3 | alert, e2e | failed | #7495 (open) | 2 | e2e (1)cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded" Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-cvz2rwgtfkl8/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-cvz2rwgtfkl8/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failedalert (1)alert [infra] Cosmos DB Normalized RU Consumption High - <cosmos-account> fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T16:36:18Z Ended: 2026-10-13T16:55:14Z Severity: Sev3 Description: Cosmos DB normalized RU consumption is above 70% averaged over a 15-minute window, evaluated every minute. Investigate workload patterns or increase provisioned throughput. https://learn.microsoft.com/azure/cosmos-db/monitor-normalized-request-units |
|
| 2h 33m 4s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
westus3 | alert | failed | #7469 (merged) | 1 | alert (1)alert [svc] BackendControllerQueueDepthHigh fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T16:31:51Z Ended: 2026-10-13T16:42:49Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8033792-svc", component="backend", name="clusterresources", region="westus3", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. |
|
| 2h 33m 39s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | centralus | alert | failed | #7451 (open) | 1 | alert (1)alert [svc] BackendControllerQueueDepthHigh fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T16:29:41Z Ended: 2026-10-13T16:48:44Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9600128-svc", component="backend", name="actualhostedcluster", region="centralus", severity="warning" Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. |
|
| 2h 40m 6s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
westus3 | alert, e2e | failed | #7485 (merged) | 4 | e2e (1)ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:89]: failed to create HCP cluster for kusto logs test
Unexpected error:
<*fmt.wrapError | 0x34adff672b00>:
failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-5xtjf6plnvzj" to finish creating: GET https://rp.j9411968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a54bc6-22fe-435f-b062-cfb9be4ee7f5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a54bc6-22fe-435f-b062-cfb9be4ee7f5",
"name": "15a54bc6-22fe-435f-b062-cfb9be4ee7f5",
"status": "Failed",
"startTime": "2026-10-13T16:18:20.725051746Z",
"endTime": "2026-10-13T16:47:20.13643423Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9411968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a54bc6-22fe-435f-b062-cfb9be4ee7f5
correlation ID: 1e394c4c-0ffa-4d52-86b7-718f2997f7ad
client request ID: 573494d7-32aa-472d-997a-4bd8db38dacc
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:89]: failed to create HCP cluster for kusto logs test
Unexpected error:
<*fmt.wrapError | 0x34adff672b00>:
failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-5xtjf6plnvzj" to finish creating: GET https://rp.j9411968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a54bc6-22fe-435f-b062-cfb9be4ee7f5
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a54bc6-22fe-435f-b062-cfb9be4ee7f5",
"name": "15a54bc6-22fe-435f-b062-cfb9be4ee7f5",
"status": "Failed",
"startTime": "2026-10-13T16:18:20.725051746Z",
"endTime": "2026-10-13T16:47:20.13643423Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9411968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/15a54bc6-22fe-435f-b062-cfb9be4ee7f5
correlation ID: 1e394c4c-0ffa-4d52-86b7-718f2997f7ad
client request ID: 573494d7-32aa-472d-997a-4bd8db38dacc
occurredalert (3)alert [hcp] KubeDeploymentReplicasMismatch fired Show raw failurealert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T17:24:10Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9411968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.31:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9411968/providers/microsoft.monitor/accounts/hcps-j9411968", namespace="ocm-arohcpci01-2te0nj9h3askcfok1isadggsor76sejq-x1r3f9l2r1b0r9d", pod="arohcp-monitor-kube-state-metrics-7988dc648c-dj7ht", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2te0nj9h3askcfok1isadggsor76sejq-x1r3f9l2r1b0r9d/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-13T17:24:10Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9411968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.65.31:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9411968/providers/microsoft.monitor/accounts/hcps-j9411968", namespace="ocm-arohcpci01-2te0nj9h3askcfok1isadggsor76sejq-x1r3f9l2r1b0r9d", pod="arohcp-monitor-kube-state-metrics-7988dc648c-dj7ht", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2te0nj9h3askcfok1isadggsor76sejq-x1r3f9l2r1b0r9d/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes. alert [hcp] KubePodCrashLooping fired Show raw failurealert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-13T16:51:06Z Ended: 2026-10-13T17:10:10Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j9411968-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.192:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9411968/providers/microsoft.monitor/accounts/hcps-j9411968", namespace="ocm-arohcpci01-2te0njd9raq4236kje9rhni2tjsaju6i-s2p4w6o3n3b4d5j", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0749a532-89d2-4bdf-99d2-f75e8cbffe37" Description: Pod ocm-arohcpci01-2te0njd9raq4236kje9rhni2tjsaju6i-s2p4w6o3n3b4d5j/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-13T16:51:06Z Ended: 2026-10-13T17:10:10Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j9411968-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.192:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9411968/providers/microsoft.monitor/accounts/hcps-j9411968", namespace="ocm-arohcpci01-2te0njd9raq4236kje9rhni2tjsaju6i-s2p4w6o3n3b4d5j", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="0749a532-89d2-4bdf-99d2-f75e8cbffe37" Description: Pod ocm-arohcpci01-2te0njd9raq4236kje9rhni2tjsaju6i-s2p4w6o3n3b4d5j/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff"). alert [hcp] KubeStatefulSetReplicasMismatch fired Show raw failurealert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-13T16:48:12Z Ended: 2026-10-13T17:05:12Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9411968-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.192:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9411968/providers/microsoft.monitor/accounts/hcps-j9411968", namespace="ocm-arohcpci01-2te0njd9raq4236kje9rhni2tjsaju6i-s2p4w6o3n3b4d5j", pod="arohcp-monitor-kube-state-metrics-7988dc648c-w2fmt", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2te0njd9raq4236kje9rhni2tjsaju6i-s2p4w6o3n3b4d5j/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-13T16:49:11Z Ended: 2026-10-13T17:06:14Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9411968-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.192:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9411968/providers/microsoft.monitor/accounts/hcps-j9411968", namespace="ocm-arohcpci01-2te0njd9raq4236kje9rhni2tjsaju6i-s2p4w6o3n3b4d5j", pod="arohcp-monitor-kube-state-metrics-7988dc648c-w2fmt", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2te0njd9raq4236kje9rhni2tjsaju6i-s2p4w6o3n3b4d5j/etcd has not matched the expected number of replicas for longer than 15 minutes. |
|
| 2h 25m 48s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
westus3 | unknown | failed | #6746 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| 2h 50m 27s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
westus3 | e2e | failed | #7480 (merged) | 1 | e2e (1)failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:139]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x1ba7ee626c00>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-jk8fzdt4b4dc" to finish creating: GET https://rp.j5689344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8fa1502c-ea74-497a-b801-49d332ad2c8c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8fa1502c-ea74-497a-b801-49d332ad2c8c",
"name": "8fa1502c-ea74-497a-b801-49d332ad2c8c",
"status": "Failed",
"startTime": "2026-10-13T15:47:39.824514687Z",
"endTime": "2026-10-13T16:16:43.528270227Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-kv8bw-nb989: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-jk8fzdt4b4dc--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-kv8bw-nb989 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-kv8bw-nb989: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-jk8fzdt4b4dc--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-kv8bw-nb989 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5689344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8fa1502c-ea74-497a-b801-49d332ad2c8c
correlation ID: 4064747a-e26a-4fb7-a7aa-4d63a3fd0e24
client request ID: 6ee5609d-a8f6-47ef-be97-c1455bdfadcf
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:139]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0x1ba7ee626c00>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-jk8fzdt4b4dc" to finish creating: GET https://rp.j5689344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8fa1502c-ea74-497a-b801-49d332ad2c8c
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8fa1502c-ea74-497a-b801-49d332ad2c8c",
"name": "8fa1502c-ea74-497a-b801-49d332ad2c8c",
"status": "Failed",
"startTime": "2026-10-13T15:47:39.824514687Z",
"endTime": "2026-10-13T16:16:43.528270227Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-kv8bw-nb989: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-jk8fzdt4b4dc--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-kv8bw-nb989 to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-kv8bw-nb989: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-jk8fzdt4b4dc--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-kv8bw-nb989 to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j5689344.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8fa1502c-ea74-497a-b801-49d332ad2c8c
correlation ID: 4064747a-e26a-4fb7-a7aa-4d63a3fd0e24
client request ID: 6ee5609d-a8f6-47ef-be97-c1455bdfadcf |
|
| 2h 55m 17s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | westus3 | n/a | passed | #7493 (open) | 0 | n/a | |
| 1h 32m 20s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
westus3 | unknown | failed | #7397 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| 38m 4s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | centralus | unknown | failed | #7493 (open) | 1 | non-artifact-backed failure (no junit artifacts) | |
| 2h 27m 35s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | westus3 | n/a | passed | #7370 (open) | 0 | n/a | |
| 2h 25m 37s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
westus3 | n/a | passed | #7492 (merged) | 0 | n/a | |
| 2h 17m 43s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | alert, e2e | failed | #7488 (merged) | 2 | e2e (1)ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x28b03e9a6680>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-69pgmlxbh7vh" to finish creating: GET https://rp.j4018048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b2e67201-af0c-42da-b508-f40f254f87c7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b2e67201-af0c-42da-b508-f40f254f87c7",
"name": "b2e67201-af0c-42da-b508-f40f254f87c7",
"status": "Failed",
"startTime": "2026-10-13T14:11:58.76773642Z",
"endTime": "2026-10-13T14:41:02.859999025Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d0b5y3p4k9a2p9w-np-1-wtm4z-m5q6d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-69pgmlxbh7vh--managed/providers/Microsoft.Compute/virtualMachines/d0b5y3p4k9a2p9w-np-1-wtm4z-m5q6d to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d0b5y3p4k9a2p9w-np-1-wtm4z-m5q6d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-69pgmlxbh7vh--managed/providers/Microsoft.Compute/virtualMachines/d0b5y3p4k9a2p9w-np-1-wtm4z-m5q6d to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4018048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b2e67201-af0c-42da-b508-f40f254f87c7
correlation ID: c7972c9d-9ddb-4652-a9ac-8fbcdb522c95
client request ID: e9e05827-93d6-4587-bf66-ed9dc69c4222
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x28b03e9a6680>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-69pgmlxbh7vh" to finish creating: GET https://rp.j4018048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b2e67201-af0c-42da-b508-f40f254f87c7
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b2e67201-af0c-42da-b508-f40f254f87c7",
"name": "b2e67201-af0c-42da-b508-f40f254f87c7",
"status": "Failed",
"startTime": "2026-10-13T14:11:58.76773642Z",
"endTime": "2026-10-13T14:41:02.859999025Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine d0b5y3p4k9a2p9w-np-1-wtm4z-m5q6d: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-69pgmlxbh7vh--managed/providers/Microsoft.Compute/virtualMachines/d0b5y3p4k9a2p9w-np-1-wtm4z-m5q6d to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine d0b5y3p4k9a2p9w-np-1-wtm4z-m5q6d: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-69pgmlxbh7vh--managed/providers/Microsoft.Compute/virtualMachines/d0b5y3p4k9a2p9w-np-1-wtm4z-m5q6d to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4018048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b2e67201-af0c-42da-b508-f40f254f87c7
correlation ID: c7972c9d-9ddb-4652-a9ac-8fbcdb522c95
client request ID: e9e05827-93d6-4587-bf66-ed9dc69c4222
occurredalert (1)alert [svc] LeaderElectionLeaseStale fired Show raw failurealert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T15:08:37Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j4018048-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.52:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4018048/providers/microsoft.monitor/accounts/services-j4018048", namespace="klusterlet-2tdunktpkavtvflvbc7rdslt6p35hl1d", pod="arohcp-monitor-kube-state-metrics-7988dc648c-kq6qh", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2tdunktpkavtvflvbc7rdslt6p35hl1d on cluster ci01-j4018048-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. |
|
| 17m 13s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | n/a | unknown | failed | #7477 (closed) | 1 | non-artifact-backed failure (no junit artifacts) | |
| 2h 16m 39s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | centralus | alert | failed | #7441 (open) | 1 | alert (1)alert [svc] LeaderElectionLeaseStale fired Show raw failurealert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T14:30:35Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j4851584-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.252:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4851584/providers/microsoft.monitor/accounts/services-j4851584", namespace="klusterlet-2tdu5qet089m4vq4jt27iijio79srvjo", pod="arohcp-monitor-kube-state-metrics-7988dc648c-fgb47", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2tdu5qet089m4vq4jt27iijio79srvjo on cluster ci01-j4851584-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. |
|
| 1h 43m 4s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
westus3 | unknown | failed | #7469 (merged) | 1 | non-artifact-backed failure (no junit artifacts) | |
| 2h 18m 31s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | westus3 | e2e | failed | #6920 (open) | 1 | e2e (1)ERROR CODE: DeploymentFailed; detail code InvalidTemplate; detail message Unable to process template language expressions for resource '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Authorization/roleAssignments/<uuid>' at line '<line>' and column '<column>'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'; provider Microsoft.Authorization Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: failed to create customer resources for autoscaling cluster
Unexpected error:
<*fmt.wrapError | 0x3398cf888a80>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-autoscaling-hcp-cluster-qzrlv9" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-autoscaling-hcp-cluster-qzrlv9/operationStatuses/08584097109520399682
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-cml7gjf827gp/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-e4ixwyxsfjnje\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"InvalidTemplate\",\r\n \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-cml7gjf827gp/providers/Microsoft.Authorization/roleAssignments/adc91f81-bb61-5a2b-b4cc-0e8c1c460b92' at line '1' and column '27135'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n \"additionalInfo\": [\r\n {\r\n \"type\": \"TemplateViolation\",\r\n \"info\": {\r\n \"lineNumber\": 1,\r\n \"linePosition\": 27135,\r\n \"path\": \"\"\r\n }\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-autoscaling-hcp-cluster-qzrlv9/operationStatuses/08584097109520399682
correlation ID: 9860797c-226a-442a-b3bd-c7b7c62b766b
client request ID: unavailable
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: failed to create customer resources for autoscaling cluster
Unexpected error:
<*fmt.wrapError | 0x3398cf888a80>:
failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-autoscaling-hcp-cluster-qzrlv9" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-autoscaling-hcp-cluster-qzrlv9/operationStatuses/08584097109520399682
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeploymentFailed
--------------------------------------------------------------------------------
{
"status": "Failed",
"error": {
"code": "DeploymentFailed",
"message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
"details": [
{
"code": "Conflict",
"message": "{\r\n \"status\": \"Failed\",\r\n \"error\": {\r\n \"code\": \"ResourceDeploymentFailure\",\r\n \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n \"details\": [\r\n {\r\n \"code\": \"DeploymentFailed\",\r\n \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-cml7gjf827gp/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-e4ixwyxsfjnje\",\r\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n \"details\": [\r\n {\r\n \"code\": \"InvalidTemplate\",\r\n \"message\": \"Unable to process template language expressions for resource '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-cml7gjf827gp/providers/Microsoft.Authorization/roleAssignments/adc91f81-bb61-5a2b-b4cc-0e8c1c460b92' at line '1' and column '27135'. 'Language expression property 'principalId' can't be evaluated, the referencing object is null.'\",\r\n \"additionalInfo\": [\r\n {\r\n \"type\": \"TemplateViolation\",\r\n \"info\": {\r\n \"lineNumber\": 1,\r\n \"linePosition\": 27135,\r\n \"path\": \"\"\r\n }\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n ]\r\n }\r\n}"
}
]
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-autoscaling-hcp-cluster-qzrlv9/operationStatuses/08584097109520399682
correlation ID: 9860797c-226a-442a-b3bd-c7b7c62b766b
client request ID: unavailable
occurred |
|
| 2h 24m 59s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | westus3 | alert, e2e | failed | #7463 (open) | 4 | e2e (1)VerifyCloudNetworkConfigEgressIP(<ip>) failed: EgressIP "e2e-cloud-network-config-test" was never assigned to a node: timed out waiting for EgressIP "e2e-cloud-network-config-test" to be assigned to a node: EgressIP "e2e-cloud-network-config-test" is not yet Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_operators.go:163]: failed to verify cluster operators on cluster "cluster-operators-qwqhtt"
Unexpected error:
<*errors.joinError | 0x275bb568108>:
VerifyCloudNetworkConfigEgressIP(10.0.0.6) failed: EgressIP "e2e-cloud-network-config-test" was never assigned to a node: timed out waiting for EgressIP "e2e-cloud-network-config-test" to be assigned to a node: EgressIP "e2e-cloud-network-config-test" is not yet assigned to a node
{
errs: [
<*fmt.wrapError | 0x275bb4fe480>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.6) failed: EgressIP \"e2e-cloud-network-config-test\" was never assigned to a node: timed out waiting for EgressIP \"e2e-cloud-network-config-test\" to be assigned to a node: EgressIP \"e2e-cloud-network-config-test\" is not yet assigned to a node",
err: <*fmt.wrapError | 0x275baa78f20>{
msg: "EgressIP \"e2e-cloud-network-config-test\" was never assigned to a node: timed out waiting for EgressIP \"e2e-cloud-network-config-test\" to be assigned to a node: EgressIP \"e2e-cloud-network-config-test\" is not yet assigned to a node",
err: <*fmt.wrapError | 0x275baa78d80>{
msg: "timed out waiting for EgressIP \"e2e-cloud-network-config-test\" to be assigned to a node: EgressIP \"e2e-cloud-network-config-test\" is not yet assigned to a node",
err: <*errors.errorString | 0x275bb7584c0>{
s: "EgressIP \"e2e-cloud-network-config-test\" is not yet assigned to a node",
},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_operators.go:163]: failed to verify cluster operators on cluster "cluster-operators-qwqhtt"
Unexpected error:
<*errors.joinError | 0x275bb568108>:
VerifyCloudNetworkConfigEgressIP(10.0.0.6) failed: EgressIP "e2e-cloud-network-config-test" was never assigned to a node: timed out waiting for EgressIP "e2e-cloud-network-config-test" to be assigned to a node: EgressIP "e2e-cloud-network-config-test" is not yet assigned to a node
{
errs: [
<*fmt.wrapError | 0x275bb4fe480>{
msg: "VerifyCloudNetworkConfigEgressIP(10.0.0.6) failed: EgressIP \"e2e-cloud-network-config-test\" was never assigned to a node: timed out waiting for EgressIP \"e2e-cloud-network-config-test\" to be assigned to a node: EgressIP \"e2e-cloud-network-config-test\" is not yet assigned to a node",
err: <*fmt.wrapError | 0x275baa78f20>{
msg: "EgressIP \"e2e-cloud-network-config-test\" was never assigned to a node: timed out waiting for EgressIP \"e2e-cloud-network-config-test\" to be assigned to a node: EgressIP \"e2e-cloud-network-config-test\" is not yet assigned to a node",
err: <*fmt.wrapError | 0x275baa78d80>{
msg: "timed out waiting for EgressIP \"e2e-cloud-network-config-test\" to be assigned to a node: EgressIP \"e2e-cloud-network-config-test\" is not yet assigned to a node",
err: <*errors.errorString | 0x275bb7584c0>{
s: "EgressIP \"e2e-cloud-network-config-test\" is not yet assigned to a node",
},
},
},
},
],
}
occurredalert (3)alert [svc] BackendControllerQueueDepthHigh fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T13:04:07Z Ended: 2026-10-13T13:24:08Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9120896-svc", component="backend", name="actualhostedcluster", region="westus3", severity="warning" Description: Backend controller workqueue actualhostedcluster has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert [hcp] KubePodNotReady fired Show raw failurealert fired 268 time(s) (214 unknown, 54 known) Firing 1: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:24:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-4p8nw", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-4p8nw has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:21:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-mth8b", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-mth8b has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-kfrcc", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-kfrcc has been in a non-ready state for longer than 5 minutes. Firing 4: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:21:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-6mmc5", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-6mmc5 has been in a non-ready state for longer than 5 minutes. Firing 5: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-gvsss", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-gvsss has been in a non-ready state for longer than 5 minutes. Firing 6: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-cvw52", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-cvw52 has been in a non-ready state for longer than 5 minutes. Firing 7: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-64j6b", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-64j6b has been in a non-ready state for longer than 5 minutes. Firing 8: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:21:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-fk5mc", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-fk5mc has been in a non-ready state for longer than 5 minutes. Firing 9: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:21:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-htwvx", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-htwvx has been in a non-ready state for longer than 5 minutes. Firing 10: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-xzksr", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-xzksr has been in a non-ready state for longer than 5 minutes. Firing 11: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:21:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-xd6hs", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-xd6hs has been in a non-ready state for longer than 5 minutes. Firing 12: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-qwtqs", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-qwtqs has been in a non-ready state for longer than 5 minutes. Firing 13: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-cd628", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-cd628 has been in a non-ready state for longer than 5 minutes. Firing 14: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:24:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-5lhgf", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-5lhgf has been in a non-ready state for longer than 5 minutes. Firing 15: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-pgq56", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-pgq56 has been in a non-ready state for longer than 5 minutes. Firing 16: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:24:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-hvd9c", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-hvd9c has been in a non-ready state for longer than 5 minutes. Firing 17: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-v8qst", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-v8qst has been in a non-ready state for longer than 5 minutes. Firing 18: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-mslbg", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-mslbg has been in a non-ready state for longer than 5 minutes. Firing 19: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-fpk2l", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-fpk2l has been in a non-ready state for longer than 5 minutes. Firing 20: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:21:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-rfn8f", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-rfn8f has been in a non-ready state for longer than 5 minutes. Firing 21: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-nm4ns", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-nm4ns has been in a non-ready state for longer than 5 minutes. Firing 22: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-9vjkc", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-9vjkc has been in a non-ready state for longer than 5 minutes. Firing 23: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-krtvh", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-krtvh has been in a non-ready state for longer than 5 minutes. Firing 24: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-ckztq", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-ckztq has been in a non-ready state for longer than 5 minutes. Firing 25: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:24:35Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-jp2dp", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-jp2dp has been in a non-ready state for longer than 5 minutes. Firing 26: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-w6nlz", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-w6nlz has been in a non-ready state for longer than 5 minutes. Firing 27: State: Resolved Started: 2026-10-13T13:07:29Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-xlqmw", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-xlqmw has been in a non-ready state for longer than 5 minutes. Firing 28: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:24:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-84j5z", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-84j5z has been in a non-ready state for longer than 5 minutes. Firing 29: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-mpcxc", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-mpcxc has been in a non-ready state for longer than 5 minutes. Firing 30: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-llsls", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-llsls has been in a non-ready state for longer than 5 minutes. Firing 31: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-2z42c", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-2z42c has been in a non-ready state for longer than 5 minutes. Firing 32: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-42tsp", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-42tsp has been in a non-ready state for longer than 5 minutes. Firing 33: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-5w88x", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-5w88x has been in a non-ready state for longer than 5 minutes. Firing 34: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:24:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-mmcn9", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-mmcn9 has been in a non-ready state for longer than 5 minutes. Firing 35: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:25:35Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-w8l2m", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-w8l2m has been in a non-ready state for longer than 5 minutes. Firing 36: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-c9bzq", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-c9bzq has been in a non-ready state for longer than 5 minutes. Firing 37: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:24:35Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-dqskl", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-dqskl has been in a non-ready state for longer than 5 minutes. Firing 38: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:24:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-2244t", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-2244t has been in a non-ready state for longer than 5 minutes. Firing 39: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-wjptp", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-wjptp has been in a non-ready state for longer than 5 minutes. Firing 40: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-tjhrj", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-tjhrj has been in a non-ready state for longer than 5 minutes. Firing 41: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:21:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-646s4", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-646s4 has been in a non-ready state for longer than 5 minutes. Firing 42: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:21:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-hn9r4", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-hn9r4 has been in a non-ready state for longer than 5 minutes. Firing 43: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-kssd5", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-kssd5 has been in a non-ready state for longer than 5 minutes. Firing 44: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:24:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-2wxwt", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-2wxwt has been in a non-ready state for longer than 5 minutes. Firing 45: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-5ffbw", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-5ffbw has been in a non-ready state for longer than 5 minutes. Firing 46: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-zszss", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-zszss has been in a non-ready state for longer than 5 minutes. Firing 47: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-p2jgh", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-p2jgh has been in a non-ready state for longer than 5 minutes. Firing 48: State: Resolved Started: 2026-10-13T13:07:30Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-8fgnv", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-8fgnv has been in a non-ready state for longer than 5 minutes. Firing 49: State: Resolved Started: 2026-10-13T13:07:31Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-h9gpx", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-h9gpx has been in a non-ready state for longer than 5 minutes. Firing 50: State: Resolved Started: 2026-10-13T13:07:31Z Ended: 2026-10-13T13:24:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-9dt95", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-9dt95 has been in a non-ready state for longer than 5 minutes. Firing 51: State: Resolved Started: 2026-10-13T13:07:31Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-ntx2r", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-ntx2r has been in a non-ready state for longer than 5 minutes. Firing 52: State: Resolved Started: 2026-10-13T13:07:31Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-hvjt4", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-hvjt4 has been in a non-ready state for longer than 5 minutes. Firing 53: State: Resolved Started: 2026-10-13T13:07:31Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-b9h4w", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-b9h4w has been in a non-ready state for longer than 5 minutes. Firing 54: State: Resolved Started: 2026-10-13T13:07:31Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-qzr2z", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-qzr2z has been in a non-ready state for longer than 5 minutes. Firing 55: State: Resolved Started: 2026-10-13T13:07:31Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-zh7v8", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-zh7v8 has been in a non-ready state for longer than 5 minutes. Firing 56: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-zxnfn", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-zxnfn has been in a non-ready state for longer than 5 minutes. Firing 57: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-ccvlz", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-ccvlz has been in a non-ready state for longer than 5 minutes. Firing 58: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-5sffd", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-5sffd has been in a non-ready state for longer than 5 minutes. Firing 59: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-sn9tj", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-sn9tj has been in a non-ready state for longer than 5 minutes. Firing 60: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:21:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-kv44x", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-kv44x has been in a non-ready state for longer than 5 minutes. Firing 61: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-5jlm5", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-5jlm5 has been in a non-ready state for longer than 5 minutes. Firing 62: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-k8g2p", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-k8g2p has been in a non-ready state for longer than 5 minutes. Firing 63: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:21:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-dx2bl", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-dx2bl has been in a non-ready state for longer than 5 minutes. Firing 64: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-7jxb4", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-7jxb4 has been in a non-ready state for longer than 5 minutes. Firing 65: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-nfwzr", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-nfwzr has been in a non-ready state for longer than 5 minutes. Firing 66: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-9vn2r", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-9vn2r has been in a non-ready state for longer than 5 minutes. Firing 67: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:24:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-fp94v", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-fp94v has been in a non-ready state for longer than 5 minutes. Firing 68: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:21:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-7tgzd", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-7tgzd has been in a non-ready state for longer than 5 minutes. Firing 69: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-rj5rm", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-rj5rm has been in a non-ready state for longer than 5 minutes. Firing 70: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-4vxxx", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-4vxxx has been in a non-ready state for longer than 5 minutes. Firing 71: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-xd6zz", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-xd6zz has been in a non-ready state for longer than 5 minutes. Firing 72: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:24:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-kpw8t", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-kpw8t has been in a non-ready state for longer than 5 minutes. Firing 73: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-8kw4b", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-8kw4b has been in a non-ready state for longer than 5 minutes. Firing 74: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-nrhmx", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-nrhmx has been in a non-ready state for longer than 5 minutes. Firing 75: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-jz6bl", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-jz6bl has been in a non-ready state for longer than 5 minutes. Firing 76: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:24:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-mqw5p", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-mqw5p has been in a non-ready state for longer than 5 minutes. Firing 77: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-rbmn4", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-rbmn4 has been in a non-ready state for longer than 5 minutes. Firing 78: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-tf5n4", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-tf5n4 has been in a non-ready state for longer than 5 minutes. Firing 79: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:21:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-khsng", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-khsng has been in a non-ready state for longer than 5 minutes. Firing 80: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-8tmzd", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-8tmzd has been in a non-ready state for longer than 5 minutes. Firing 81: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:35Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-5skgc", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-5skgc has been in a non-ready state for longer than 5 minutes. Firing 82: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-59ktc", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-59ktc has been in a non-ready state for longer than 5 minutes. Firing 83: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:24:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-z9qk6", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-z9qk6 has been in a non-ready state for longer than 5 minutes. Firing 84: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:24:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-6wcbs", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-6wcbs has been in a non-ready state for longer than 5 minutes. Firing 85: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-2vc6m", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-2vc6m has been in a non-ready state for longer than 5 minutes. Firing 86: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:24:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-4xw4t", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-4xw4t has been in a non-ready state for longer than 5 minutes. Firing 87: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-88nm4", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-88nm4 has been in a non-ready state for longer than 5 minutes. Firing 88: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:24:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-v8bjr", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-v8bjr has been in a non-ready state for longer than 5 minutes. Firing 89: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:21:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-6p6dh", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-6p6dh has been in a non-ready state for longer than 5 minutes. Firing 90: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-fmbv8", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-fmbv8 has been in a non-ready state for longer than 5 minutes. Firing 91: State: Resolved Started: 2026-10-13T13:07:32Z Ended: 2026-10-13T13:24:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-hrnst", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-hrnst has been in a non-ready state for longer than 5 minutes. Firing 92: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-cnb9k", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-cnb9k has been in a non-ready state for longer than 5 minutes. Firing 93: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-hrqjl", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-hrqjl has been in a non-ready state for longer than 5 minutes. Firing 94: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-b4llx", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-b4llx has been in a non-ready state for longer than 5 minutes. Firing 95: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-d65nj", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-d65nj has been in a non-ready state for longer than 5 minutes. Firing 96: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-kfpll", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-kfpll has been in a non-ready state for longer than 5 minutes. Firing 97: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-rr42p", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-rr42p has been in a non-ready state for longer than 5 minutes. Firing 98: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-kskk4", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-kskk4 has been in a non-ready state for longer than 5 minutes. Firing 99: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-rwcpc", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-rwcpc has been in a non-ready state for longer than 5 minutes. Firing 100: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:24:35Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-4vtl9", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-4vtl9 has been in a non-ready state for longer than 5 minutes. Firing 101: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-vfdsj", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-vfdsj has been in a non-ready state for longer than 5 minutes. Firing 102: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:24:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-zdlpc", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-zdlpc has been in a non-ready state for longer than 5 minutes. Firing 103: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-22pkh", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-22pkh has been in a non-ready state for longer than 5 minutes. Firing 104: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-b7v7h", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-b7v7h has been in a non-ready state for longer than 5 minutes. Firing 105: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-4dtxs", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-4dtxs has been in a non-ready state for longer than 5 minutes. Firing 106: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-lqsrv", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-lqsrv has been in a non-ready state for longer than 5 minutes. Firing 107: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:21:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-5kmv8", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-5kmv8 has been in a non-ready state for longer than 5 minutes. Firing 108: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-rdfmh", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-rdfmh has been in a non-ready state for longer than 5 minutes. Firing 109: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:21:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-dr2lh", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-dr2lh has been in a non-ready state for longer than 5 minutes. Firing 110: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:21:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-p685x", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-p685x has been in a non-ready state for longer than 5 minutes. Firing 111: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-dw2mp", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-dw2mp has been in a non-ready state for longer than 5 minutes. Firing 112: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:24:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-657fn", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-657fn has been in a non-ready state for longer than 5 minutes. Firing 113: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-jfq9s", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-jfq9s has been in a non-ready state for longer than 5 minutes. Firing 114: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-k4tcm", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-k4tcm has been in a non-ready state for longer than 5 minutes. Firing 115: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:24:35Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-t8bkn", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-t8bkn has been in a non-ready state for longer than 5 minutes. Firing 116: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-tx7lr", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-tx7lr has been in a non-ready state for longer than 5 minutes. Firing 117: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:24:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-6j7bp", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-6j7bp has been in a non-ready state for longer than 5 minutes. Firing 118: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-rx2xw", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-rx2xw has been in a non-ready state for longer than 5 minutes. Firing 119: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-5ljqm", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-5ljqm has been in a non-ready state for longer than 5 minutes. Firing 120: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-2tfvd", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-2tfvd has been in a non-ready state for longer than 5 minutes. Firing 121: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-8n2np", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-8n2np has been in a non-ready state for longer than 5 minutes. Firing 122: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:21:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-cfdxv", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-cfdxv has been in a non-ready state for longer than 5 minutes. Firing 123: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:35Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-bprfj", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-bprfj has been in a non-ready state for longer than 5 minutes. Firing 124: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-hnr8c", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-hnr8c has been in a non-ready state for longer than 5 minutes. Firing 125: State: Resolved Started: 2026-10-13T13:07:33Z Ended: 2026-10-13T13:21:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-l9z76", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-l9z76 has been in a non-ready state for longer than 5 minutes. Firing 126: State: Resolved Started: 2026-10-13T13:07:34Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-r9dl8", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-r9dl8 has been in a non-ready state for longer than 5 minutes. Firing 127: State: Resolved Started: 2026-10-13T13:07:34Z Ended: 2026-10-13T13:21:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-8sjtl", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-8sjtl has been in a non-ready state for longer than 5 minutes. Firing 128: State: Resolved Started: 2026-10-13T13:07:34Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-hmhnl", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-hmhnl has been in a non-ready state for longer than 5 minutes. Firing 129: State: Resolved Started: 2026-10-13T13:07:34Z Ended: 2026-10-13T13:24:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-v79bm", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-v79bm has been in a non-ready state for longer than 5 minutes. Firing 130: State: Resolved Started: 2026-10-13T13:07:34Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-zwghk", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-zwghk has been in a non-ready state for longer than 5 minutes. Firing 131: State: Resolved Started: 2026-10-13T13:07:34Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-fj72b", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-fj72b has been in a non-ready state for longer than 5 minutes. Firing 132: State: Resolved Started: 2026-10-13T13:07:34Z Ended: 2026-10-13T13:21:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-ppzvj", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-ppzvj has been in a non-ready state for longer than 5 minutes. Firing 133: State: Resolved Started: 2026-10-13T13:07:34Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-jvdj2", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-jvdj2 has been in a non-ready state for longer than 5 minutes. Firing 134: State: Resolved Started: 2026-10-13T13:07:35Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-fp2ch", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-fp2ch has been in a non-ready state for longer than 5 minutes. Firing 135: State: Resolved Started: 2026-10-13T13:07:35Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-gbcsh", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-gbcsh has been in a non-ready state for longer than 5 minutes. Firing 136: State: Resolved Started: 2026-10-13T13:07:35Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-7lpps", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-7lpps has been in a non-ready state for longer than 5 minutes. Firing 137: State: Resolved Started: 2026-10-13T13:07:35Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-dprms", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-dprms has been in a non-ready state for longer than 5 minutes. Firing 138: State: Resolved Started: 2026-10-13T13:07:35Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-9w7bz", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-9w7bz has been in a non-ready state for longer than 5 minutes. Firing 139: State: Resolved Started: 2026-10-13T13:07:35Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-45l4z", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-45l4z has been in a non-ready state for longer than 5 minutes. Firing 140: State: Resolved Started: 2026-10-13T13:07:36Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-s625c", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-s625c has been in a non-ready state for longer than 5 minutes. Firing 141: State: Resolved Started: 2026-10-13T13:07:36Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-629qx", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-629qx has been in a non-ready state for longer than 5 minutes. Firing 142: State: Resolved Started: 2026-10-13T13:07:36Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-qb6nb", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-qb6nb has been in a non-ready state for longer than 5 minutes. Firing 143: State: Resolved Started: 2026-10-13T13:07:36Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-7nmbz", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-7nmbz has been in a non-ready state for longer than 5 minutes. Firing 144: State: Resolved Started: 2026-10-13T13:07:36Z Ended: 2026-10-13T13:24:35Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-pthc4", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-pthc4 has been in a non-ready state for longer than 5 minutes. Firing 145: State: Resolved Started: 2026-10-13T13:07:36Z Ended: 2026-10-13T13:24:35Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-v8p7j", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-v8p7j has been in a non-ready state for longer than 5 minutes. Firing 146: State: Resolved Started: 2026-10-13T13:07:36Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-xzw8c", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-xzw8c has been in a non-ready state for longer than 5 minutes. Firing 147: State: Resolved Started: 2026-10-13T13:07:37Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-lchx8", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-lchx8 has been in a non-ready state for longer than 5 minutes. Firing 148: State: Resolved Started: 2026-10-13T13:07:37Z Ended: 2026-10-13T13:24:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-vpjc5", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-vpjc5 has been in a non-ready state for longer than 5 minutes. Firing 149: State: Resolved Started: 2026-10-13T13:07:37Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-xxtjb", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-xxtjb has been in a non-ready state for longer than 5 minutes. Firing 150: State: Resolved Started: 2026-10-13T13:07:37Z Ended: 2026-10-13T13:21:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-k8d2n", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-k8d2n has been in a non-ready state for longer than 5 minutes. Firing 151: State: Resolved Started: 2026-10-13T13:07:37Z Ended: 2026-10-13T13:24:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-gxl6f", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-gxl6f has been in a non-ready state for longer than 5 minutes. Firing 152: State: Resolved Started: 2026-10-13T13:07:37Z Ended: 2026-10-13T13:21:29Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-86q5x", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-86q5x has been in a non-ready state for longer than 5 minutes. Firing 153: State: Resolved Started: 2026-10-13T13:07:37Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-gj2jg", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-gj2jg has been in a non-ready state for longer than 5 minutes. Firing 154: State: Resolved Started: 2026-10-13T13:07:37Z Ended: 2026-10-13T13:25:35Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-f2jnj", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-f2jnj has been in a non-ready state for longer than 5 minutes. Firing 155: State: Resolved Started: 2026-10-13T13:07:37Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-4kt8w", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-4kt8w has been in a non-ready state for longer than 5 minutes. Firing 156: State: Resolved Started: 2026-10-13T13:07:37Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-2rcrh", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-2rcrh has been in a non-ready state for longer than 5 minutes. Firing 157: State: Resolved Started: 2026-10-13T13:07:38Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-cml42", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-cml42 has been in a non-ready state for longer than 5 minutes. Firing 158: State: Resolved Started: 2026-10-13T13:07:38Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-6lvzr", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-6lvzr has been in a non-ready state for longer than 5 minutes. Firing 159: State: Resolved Started: 2026-10-13T13:07:38Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-hrjz4", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-hrjz4 has been in a non-ready state for longer than 5 minutes. Firing 160: State: Resolved Started: 2026-10-13T13:07:38Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-zs9rn", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-zs9rn has been in a non-ready state for longer than 5 minutes. Firing 161: State: Resolved Started: 2026-10-13T13:07:38Z Ended: 2026-10-13T13:21:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-4bkzc", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-4bkzc has been in a non-ready state for longer than 5 minutes. Firing 162: State: Resolved Started: 2026-10-13T13:07:38Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-s2mfz", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-s2mfz has been in a non-ready state for longer than 5 minutes. Firing 163: State: Resolved Started: 2026-10-13T13:07:38Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-qqgnn", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-qqgnn has been in a non-ready state for longer than 5 minutes. Firing 164: State: Resolved Started: 2026-10-13T13:07:39Z Ended: 2026-10-13T13:24:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-frbjb", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-frbjb has been in a non-ready state for longer than 5 minutes. Firing 165: State: Resolved Started: 2026-10-13T13:07:39Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-h78dn", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-h78dn has been in a non-ready state for longer than 5 minutes. Firing 166: State: Resolved Started: 2026-10-13T13:07:39Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-wfxjs", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-wfxjs has been in a non-ready state for longer than 5 minutes. Firing 167: State: Resolved Started: 2026-10-13T13:07:39Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-9n978", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-9n978 has been in a non-ready state for longer than 5 minutes. Firing 168: State: Resolved Started: 2026-10-13T13:07:39Z Ended: 2026-10-13T13:24:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-6k44h", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-6k44h has been in a non-ready state for longer than 5 minutes. Firing 169: State: Resolved Started: 2026-10-13T13:07:40Z Ended: 2026-10-13T13:24:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-dr7bd", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-dr7bd has been in a non-ready state for longer than 5 minutes. Firing 170: State: Resolved Started: 2026-10-13T13:07:40Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-sp5tk", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-sp5tk has been in a non-ready state for longer than 5 minutes. Firing 171: State: Resolved Started: 2026-10-13T13:07:40Z Ended: 2026-10-13T13:21:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-lrjdn", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-lrjdn has been in a non-ready state for longer than 5 minutes. Firing 172: State: Resolved Started: 2026-10-13T13:07:40Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-ffgrn", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-ffgrn has been in a non-ready state for longer than 5 minutes. Firing 173: State: Resolved Started: 2026-10-13T13:07:40Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-5jvql", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-5jvql has been in a non-ready state for longer than 5 minutes. Firing 174: State: Resolved Started: 2026-10-13T13:07:40Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-2gt9h", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-2gt9h has been in a non-ready state for longer than 5 minutes. Firing 175: State: Resolved Started: 2026-10-13T13:07:40Z Ended: 2026-10-13T13:21:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-tgwzt", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-tgwzt has been in a non-ready state for longer than 5 minutes. Firing 176: State: Resolved Started: 2026-10-13T13:07:40Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-lmzwn", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-lmzwn has been in a non-ready state for longer than 5 minutes. Firing 177: State: Resolved Started: 2026-10-13T13:07:40Z Ended: 2026-10-13T13:24:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-dtbpb", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-dtbpb has been in a non-ready state for longer than 5 minutes. Firing 178: State: Resolved Started: 2026-10-13T13:07:41Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-gkljq", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-gkljq has been in a non-ready state for longer than 5 minutes. Firing 179: State: Resolved Started: 2026-10-13T13:07:41Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-v6w29", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-v6w29 has been in a non-ready state for longer than 5 minutes. Firing 180: State: Resolved Started: 2026-10-13T13:07:41Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-7drh8", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-7drh8 has been in a non-ready state for longer than 5 minutes. Firing 181: State: Resolved Started: 2026-10-13T13:07:41Z Ended: 2026-10-13T13:21:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-vqlm7", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-vqlm7 has been in a non-ready state for longer than 5 minutes. Firing 182: State: Resolved Started: 2026-10-13T13:07:41Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-hqbqc", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-hqbqc has been in a non-ready state for longer than 5 minutes. Firing 183: State: Resolved Started: 2026-10-13T13:07:41Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-v4cx8", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-v4cx8 has been in a non-ready state for longer than 5 minutes. Firing 184: State: Resolved Started: 2026-10-13T13:07:41Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-v7rhp", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-v7rhp has been in a non-ready state for longer than 5 minutes. Firing 185: State: Resolved Started: 2026-10-13T13:07:41Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-twpt9", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-twpt9 has been in a non-ready state for longer than 5 minutes. Firing 186: State: Resolved Started: 2026-10-13T13:07:41Z Ended: 2026-10-13T13:21:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-lbp46", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-lbp46 has been in a non-ready state for longer than 5 minutes. Firing 187: State: Resolved Started: 2026-10-13T13:07:42Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-fvh9x", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-fvh9x has been in a non-ready state for longer than 5 minutes. Firing 188: State: Resolved Started: 2026-10-13T13:07:42Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-7d2tq", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-7d2tq has been in a non-ready state for longer than 5 minutes. Firing 189: State: Resolved Started: 2026-10-13T13:07:42Z Ended: 2026-10-13T13:24:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-k66gb", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-k66gb has been in a non-ready state for longer than 5 minutes. Firing 190: State: Resolved Started: 2026-10-13T13:07:42Z Ended: 2026-10-13T13:21:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-x97ps", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-x97ps has been in a non-ready state for longer than 5 minutes. Firing 191: State: Resolved Started: 2026-10-13T13:07:42Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-9v7v5", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-9v7v5 has been in a non-ready state for longer than 5 minutes. Firing 192: State: Resolved Started: 2026-10-13T13:07:42Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-l8wfz", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-l8wfz has been in a non-ready state for longer than 5 minutes. Firing 193: State: Resolved Started: 2026-10-13T13:07:42Z Ended: 2026-10-13T13:21:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-jwnn7", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-jwnn7 has been in a non-ready state for longer than 5 minutes. Firing 194: State: Resolved Started: 2026-10-13T13:07:42Z Ended: 2026-10-13T13:25:35Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-hp68j", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-hp68j has been in a non-ready state for longer than 5 minutes. Firing 195: State: Resolved Started: 2026-10-13T13:07:42Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-f2rpx", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-f2rpx has been in a non-ready state for longer than 5 minutes. Firing 196: State: Resolved Started: 2026-10-13T13:07:43Z Ended: 2026-10-13T13:24:35Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-2j9gr", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-2j9gr has been in a non-ready state for longer than 5 minutes. Firing 197: State: Resolved Started: 2026-10-13T13:07:43Z Ended: 2026-10-13T13:24:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-rbfss", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-rbfss has been in a non-ready state for longer than 5 minutes. Firing 198: State: Resolved Started: 2026-10-13T13:07:43Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-lp2nr", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-lp2nr has been in a non-ready state for longer than 5 minutes. Firing 199: State: Resolved Started: 2026-10-13T13:07:43Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-wcvm5", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-wcvm5 has been in a non-ready state for longer than 5 minutes. Firing 200: State: Resolved Started: 2026-10-13T13:07:43Z Ended: 2026-10-13T13:24:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-pb78h", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-pb78h has been in a non-ready state for longer than 5 minutes. Firing 201: State: Resolved Started: 2026-10-13T13:07:44Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-v7mnn", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-v7mnn has been in a non-ready state for longer than 5 minutes. Firing 202: State: Resolved Started: 2026-10-13T13:07:44Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-6pk5z", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-6pk5z has been in a non-ready state for longer than 5 minutes. Firing 203: State: Resolved Started: 2026-10-13T13:07:44Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-b468t", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-b468t has been in a non-ready state for longer than 5 minutes. Firing 204: State: Resolved Started: 2026-10-13T13:07:44Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-pd5l9", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-pd5l9 has been in a non-ready state for longer than 5 minutes. Firing 205: State: Resolved Started: 2026-10-13T13:07:44Z Ended: 2026-10-13T13:25:34Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-8fhdq", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-8fhdq has been in a non-ready state for longer than 5 minutes. Firing 206: State: Resolved Started: 2026-10-13T13:07:44Z Ended: 2026-10-13T13:25:32Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-fmjj6", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-fmjj6 has been in a non-ready state for longer than 5 minutes. Firing 207: State: Resolved Started: 2026-10-13T13:07:44Z Ended: 2026-10-13T13:24:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-w96fn", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-w96fn has been in a non-ready state for longer than 5 minutes. Firing 208: State: Resolved Started: 2026-10-13T13:07:44Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="network-node-identity-7ff6c58d49-29m5r", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/network-node-identity-7ff6c58d49-29m5r has been in a non-ready state for longer than 5 minutes. Firing 209: State: Resolved Started: 2026-10-13T13:07:44Z Ended: 2026-10-13T13:21:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-h8gwt", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-h8gwt has been in a non-ready state for longer than 5 minutes. Firing 210: State: Resolved Started: 2026-10-13T13:07:45Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-pdbd9", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-pdbd9 has been in a non-ready state for longer than 5 minutes. Firing 211: State: Resolved Started: 2026-10-13T13:07:45Z Ended: 2026-10-13T13:25:33Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-qhb62", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-qhb62 has been in a non-ready state for longer than 5 minutes. Firing 212: State: Resolved Started: 2026-10-13T13:07:45Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="ignition-server-proxy-96959d9bf-tnrhm", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/ignition-server-proxy-96959d9bf-tnrhm has been in a non-ready state for longer than 5 minutes. Firing 213: State: Resolved Started: 2026-10-13T13:07:45Z Ended: 2026-10-13T13:25:31Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v", pod="kube-controller-manager-7c6f78c4-99vg4", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjq0klhk58c5rtaj9flc2hi84phla-p9a4v3n6g6e5b6v/kube-controller-manager-7c6f78c4-99vg4 has been in a non-ready state for longer than 5 minutes. Firing 214: State: Resolved Started: 2026-10-13T13:07:45Z Ended: 2026-10-13T13:25:30Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j9120896-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v", pod="ovnkube-control-plane-7db599cd56-9tn2z", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tdtjv2mvp1nier7asur6or8ihbkq1og-d8q9r9d9f9e2k1v/ovnkube-control-plane-7db599cd56-9tn2z has been in a non-ready state for longer than 5 minutes. alert [svc] LeaderElectionLeaseStale fired Show raw failurealert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T13:33:58Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j9120896-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.142:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9120896/providers/microsoft.monitor/accounts/services-j9120896", namespace="klusterlet-2tdtjr3ne5vj3bi3oa6tbe1at5fpus2l", pod="arohcp-monitor-kube-state-metrics-7988dc648c-hmgwl", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2tdtjr3ne5vj3bi3oa6tbe1at5fpus2l on cluster ci01-j9120896-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. |
|
| 2h 22m 23s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | alert | failed | #7484 (merged) | 2 | alert (2)alert [svc] LeaderElectionLeaseStale fired Show raw failurealert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T13:37:23Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j2052480-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.129:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2052480/providers/microsoft.monitor/accounts/services-j2052480", namespace="klusterlet-2tdtevmc4d8jve41e86ki97n3m7mpsvq", pod="arohcp-monitor-kube-state-metrics-7988dc648c-mlbkt", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2tdtevmc4d8jve41e86ki97n3m7mpsvq on cluster ci01-j2052480-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. alert [svc] BackendControllerQueueDepthHigh fired Show raw failurealert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-13T12:51:04Z Ended: 2026-10-13T13:11:03Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2052480-svc", component="backend", name="clusterresources", region="centralus", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. |
|
| 3h 32m 48s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
westus3 | e2e | failed | #7485 (merged) | 2 | e2e (2)ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [hostedCluster] HostedCluster still exists; [resourceTeardown] tearing down pod network instances, pod networks, namespaces; [descendantResources] remaining resources: applydesires, nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
<*errors.joinError | 0x12402d3b2b10>:
resource group "rg-candidate-4-23-qt7g4z697d5b": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-ch4n9f" in resourcegroup="rg-candidate-4-23-qt7g4z697d5b" to finish deleting: GET https://rp.j4233472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/27faaa2f-8540-458b-a125-3f3536544cd1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/27faaa2f-8540-458b-a125-3f3536544cd1",
"name": "27faaa2f-8540-458b-a125-3f3536544cd1",
"status": "Failed",
"startTime": "2026-10-13T12:59:59.269440518Z",
"endTime": "2026-10-13T13:24:08.339110012Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tdtgm7u9ipde1epc2rt5pe43isp14m7 still exists (deletion dispatched at 2026-10-13T12:59:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4233472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/27faaa2f-8540-458b-a125-3f3536544cd1
correlation ID: 365de814-fd9f-4b17-8525-a2096af1cada
client request ID: 166f0483-5054-4255-b484-02e488295af5
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:302]: Unexpected error:
<*errors.joinError | 0x12402d3b2b10>:
resource group "rg-candidate-4-23-qt7g4z697d5b": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-23-ch4n9f" in resourcegroup="rg-candidate-4-23-qt7g4z697d5b" to finish deleting: GET https://rp.j4233472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/27faaa2f-8540-458b-a125-3f3536544cd1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/27faaa2f-8540-458b-a125-3f3536544cd1",
"name": "27faaa2f-8540-458b-a125-3f3536544cd1",
"status": "Failed",
"startTime": "2026-10-13T12:59:59.269440518Z",
"endTime": "2026-10-13T13:24:08.339110012Z",
"error": {
"code": "InternalServerError",
"message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2tdtgm7u9ipde1epc2rt5pe43isp14m7 still exists (deletion dispatched at 2026-10-13T12:59:59Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 3 microsoft.redhatopenshift/hcpopenshiftclusters/applydesires; [resourceTeardown] tearing down pod network instances (1), pod networks (1), namespaces (2)"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4233472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/27faaa2f-8540-458b-a125-3f3536544cd1
correlation ID: 365de814-fd9f-4b17-8525-a2096af1cada
client request ID: 166f0483-5054-4255-b484-02e488295af5
occurredERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingOnInfrastructureReady: Cluster infrastructure is still provisioning; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-w5zxw6n76pst/cluster-candidate-4-20-dnk6t4 should provision
Unexpected error:
<*fmt.wrapError | 0x4d01c384420>:
failed to create HCP cluster cluster-candidate-4-20-dnk6t4: failed waiting for cluster="cluster-candidate-4-20-dnk6t4" in resourcegroup="rg-candidate-4-20-w5zxw6n76pst" to finish creating: GET https://rp.j4233472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d8dbdcd4-fba9-45cb-9ad1-2c429fbb8dfb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d8dbdcd4-fba9-45cb-9ad1-2c429fbb8dfb",
"name": "d8dbdcd4-fba9-45cb-9ad1-2c429fbb8dfb",
"status": "Failed",
"startTime": "2026-10-13T12:39:39.471658263Z",
"endTime": "2026-10-13T13:08:48.195755915Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingOnInfrastructureReady: Cluster infrastructure is still provisioning; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4233472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d8dbdcd4-fba9-45cb-9ad1-2c429fbb8dfb
correlation ID: 6fd24ae1-3485-4cf6-8f00-56a00457a1d3
client request ID: 52d0a7ef-5194-4884-9aba-b6475a9fa220
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-w5zxw6n76pst/cluster-candidate-4-20-dnk6t4 should provision
Unexpected error:
<*fmt.wrapError | 0x4d01c384420>:
failed to create HCP cluster cluster-candidate-4-20-dnk6t4: failed waiting for cluster="cluster-candidate-4-20-dnk6t4" in resourcegroup="rg-candidate-4-20-w5zxw6n76pst" to finish creating: GET https://rp.j4233472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d8dbdcd4-fba9-45cb-9ad1-2c429fbb8dfb
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d8dbdcd4-fba9-45cb-9ad1-2c429fbb8dfb",
"name": "d8dbdcd4-fba9-45cb-9ad1-2c429fbb8dfb",
"status": "Failed",
"startTime": "2026-10-13T12:39:39.471658263Z",
"endTime": "2026-10-13T13:08:48.195755915Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: WaitingOnInfrastructureReady: Cluster infrastructure is still provisioning; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4233472.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d8dbdcd4-fba9-45cb-9ad1-2c429fbb8dfb
correlation ID: 6fd24ae1-3485-4cf6-8f00-56a00457a1d3
client request ID: 52d0a7ef-5194-4884-9aba-b6475a9fa220
occurred |
|
| 2h 14m 58s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | alert, e2e | failed | #7491 (merged) | 3 | e2e (1)cluster <resource-group>/<cluster> entered terminal state "Failed" after role assignment deployment; expected "Succeeded" Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-mnl9xxqhhnd5/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failed
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.003s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster delayed-rbac-mnl9xxqhhnd5/delayed-rbac-cluster entered terminal state "Failed" after role assignment deployment; expected "Succeeded"
Expected
<armredhatopenshifthcp.ProvisioningState>: Failed
not to equal
<armredhatopenshifthcp.ProvisioningState>: Failedalert (2)alert [svc] BackendControllerQueueDepthHigh fired Show raw failurealert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-13T12:43:21Z Ended: 2026-10-13T12:58:20Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4977792-svc", component="backend", name="csstatedump", region="centralus", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Firing 2: State: Resolved Started: 2026-10-13T12:43:21Z Ended: 2026-10-13T12:58:20Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4977792-svc", component="backend", name="clusterresources", region="centralus", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert [hcp] KubeDeploymentReplicasMismatch fired Show raw failurealert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T13:32:58Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4977792-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.113:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4977792/providers/microsoft.monitor/accounts/hcps-j4977792", namespace="ocm-arohcpci01-2tdtau6376nk4eg778am4h7lka7gfbpr-b6r1t5h3h7g0b9b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-r5hzc", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tdtau6376nk4eg778am4h7lka7gfbpr-b6r1t5h3h7g0b9b/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-13T13:32:58Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4977792-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.113:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4977792/providers/microsoft.monitor/accounts/hcps-j4977792", namespace="ocm-arohcpci01-2tdtau6376nk4eg778am4h7lka7gfbpr-b6r1t5h3h7g0b9b", pod="arohcp-monitor-kube-state-metrics-7988dc648c-r5hzc", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tdtau6376nk4eg778am4h7lka7gfbpr-b6r1t5h3h7g0b9b/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes. |
|
| 2h 19m 39s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
westus3 | e2e | failed | #7488 (merged) | 1 | e2e (1)ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x2219287fa880>:
failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-v5gf99nj48r6" to finish creating: GET https://rp.j0001920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2199d8bb-869a-41e2-acec-31983e058033
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2199d8bb-869a-41e2-acec-31983e058033",
"name": "2199d8bb-869a-41e2-acec-31983e058033",
"status": "Failed",
"startTime": "2026-10-13T11:37:06.655702009Z",
"endTime": "2026-10-13T12:06:15.607847324Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.18 is Partial (want Completed), started 8m45s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j0001920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2199d8bb-869a-41e2-acec-31983e058033
correlation ID: 436fe8be-4edd-4c43-bff6-c5ba6dee20f5
client request ID: 88121473-7bcb-4b6c-858a-daddb3fc95db
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
<*fmt.wrapError | 0x2219287fa880>:
failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-v5gf99nj48r6" to finish creating: GET https://rp.j0001920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2199d8bb-869a-41e2-acec-31983e058033
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2199d8bb-869a-41e2-acec-31983e058033",
"name": "2199d8bb-869a-41e2-acec-31983e058033",
"status": "Failed",
"startTime": "2026-10-13T11:37:06.655702009Z",
"endTime": "2026-10-13T12:06:15.607847324Z",
"error": {
"code": "DeadlineExceeded",
"message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.18 is Partial (want Completed), started 8m45s ago"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j0001920.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2199d8bb-869a-41e2-acec-31983e058033
correlation ID: 436fe8be-4edd-4c43-bff6-c5ba6dee20f5
client request ID: 88121473-7bcb-4b6c-858a-daddb3fc95db
occurred |
|
| 2h 9m 52s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
westus3 | alert | failed | #6944 (merged) | 1 | alert (1)alert [hcp] KubePodNotReady fired Show raw failurealert fired 7 time(s) (1 unknown, 6 known) Firing 1: State: Resolved Started: 2026-10-13T11:38:40Z Ended: 2026-10-13T11:44:40Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j5145472-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tds5npj51044ccirlodrkqfmqvpa3kh-t7y6j3e0f4o4f5n", pod="cluster-network-operator-78d46c7fbb-8nwxt", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tds5npj51044ccirlodrkqfmqvpa3kh-t7y6j3e0f4o4f5n/cluster-network-operator-78d46c7fbb-8nwxt has been in a non-ready state for longer than 5 minutes. |
|
| 41m 58s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
westus3 | provision | failed | #7440 (merged) | 1 | provision (1)ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url> Show raw failuretime=2026-10-13T10:18:54.749Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 description="Step infra\n Kind: ARM\n Template: templates/mgmt-infra.bicep\n Parameters: configurations/mgmt-infra.tmpl.bicepparam"
time=2026-10-13T10:18:55.539Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2
time=2026-10-13T10:18:56.816Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 deployment=fa5bd4309de472f0310387105d9534c5f4fd97370ffb7825457798baaa1e4cb1 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j0355584-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Ffa5bd4309de472f0310387105d9534c5f4fd97370ffb7825457798baaa1e4cb1
time=2026-10-13T10:20:27.529Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0355584-mgmt-2/providers/Microsoft.Resources/deployments/fa5bd4309de472f0310387105d9534c5f4fd97370ffb7825457798baaa1e4cb1/operationStatuses/08584097201492563595\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j0355584-mgmt-2/providers/Microsoft.Resources/deployments/ah-ci01-me-j0355584-2\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"VaultAlreadyExists\\\",\\r\\n \\\"message\\\": \\\"The vault name 'ah-ci01-me-j0355584-2' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n" |
|
| 2h 14m 49s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | westus3 | n/a | passed | #7490 (closed) | 0 | n/a | |
| 2h 33m 42s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
centralus | e2e | failed | #7383 (merged) | 1 | e2e (1)cluster "<cluster>" did not complete the SRE-pinned version 4.21.36 Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.005s.
cluster "sre-version-pin-z4tqzw" did not complete the SRE-pinned version 4.21.36
Expected
<string>: 4.21.36 (Partial)
to equal
<string>: 4.21.36
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.005s.
cluster "sre-version-pin-z4tqzw" did not complete the SRE-pinned version 4.21.36
Expected
<string>: 4.21.36 (Partial)
to equal
<string>: 4.21.36 |
|
| 2h 23m 54s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | n/a | passed | #7389 (merged) | 0 | n/a | |
| 2h 24m 42s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | westus3 | e2e | failed | #6920 (open) | 1 | e2e (1)ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x121182295b80>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-thbvsvb7b8fq" to finish creating: GET https://rp.j4892544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c590bff-d1a2-4a4d-8670-8ed88c881da1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c590bff-d1a2-4a4d-8670-8ed88c881da1",
"name": "6c590bff-d1a2-4a4d-8670-8ed88c881da1",
"status": "Failed",
"startTime": "2026-10-13T10:21:53.436571535Z",
"endTime": "2026-10-13T10:50:54.288659415Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m4q0a7n6v3e2h7g-np-1-ckxxw-dkp6n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-thbvsvb7b8fq--managed/providers/Microsoft.Compute/virtualMachines/m4q0a7n6v3e2h7g-np-1-ckxxw-dkp6n to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m4q0a7n6v3e2h7g-np-1-ckxxw-dkp6n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-thbvsvb7b8fq--managed/providers/Microsoft.Compute/virtualMachines/m4q0a7n6v3e2h7g-np-1-ckxxw-dkp6n to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4892544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c590bff-d1a2-4a4d-8670-8ed88c881da1
correlation ID: a74e8836-2a27-4d03-aee8-222257d39c0c
client request ID: 47ce1976-0b05-4962-8dcb-6f506b053324
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
<*fmt.wrapError | 0x121182295b80>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-thbvsvb7b8fq" to finish creating: GET https://rp.j4892544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c590bff-d1a2-4a4d-8670-8ed88c881da1
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c590bff-d1a2-4a4d-8670-8ed88c881da1",
"name": "6c590bff-d1a2-4a4d-8670-8ed88c881da1",
"status": "Failed",
"startTime": "2026-10-13T10:21:53.436571535Z",
"endTime": "2026-10-13T10:50:54.288659415Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine m4q0a7n6v3e2h7g-np-1-ckxxw-dkp6n: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-thbvsvb7b8fq--managed/providers/Microsoft.Compute/virtualMachines/m4q0a7n6v3e2h7g-np-1-ckxxw-dkp6n to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine m4q0a7n6v3e2h7g-np-1-ckxxw-dkp6n: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-thbvsvb7b8fq--managed/providers/Microsoft.Compute/virtualMachines/m4q0a7n6v3e2h7g-np-1-ckxxw-dkp6n to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j4892544.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c590bff-d1a2-4a4d-8670-8ed88c881da1
correlation ID: a74e8836-2a27-4d03-aee8-222257d39c0c
client request ID: 47ce1976-0b05-4962-8dcb-6f506b053324
occurred |
|
| 2h 25m 26s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
westus3 | n/a | passed | #7489 (merged) | 0 | n/a | |
| 2h 26m 42s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
westus3 | alert | failed | #7488 (merged) | 1 | alert (1)alert [svc] LeaderElectionLeaseStale fired Show raw failurealert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T10:15:17Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j2238464-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.6:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2238464/providers/microsoft.monitor/accounts/services-j2238464", namespace="klusterlet-2tdqe5b5src4hkonr0ghmop9gvt133es", pod="arohcp-monitor-kube-state-metrics-7988dc648c-bfr2k", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2tdqe5b5src4hkonr0ghmop9gvt133es on cluster ci01-j2238464-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. |
|
| 2h 22m 47s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
centralus | n/a | passed | #7457 (merged) | 0 | n/a | |
| 6h 45m 50s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
westus3 | alert, e2e | failed | #7485 (merged) | 2 | e2e (1)failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url> Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:139]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0xd6f1ba92440>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-tk2gb5zvpbxs" to finish creating: GET https://rp.j8911104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c1a3db7-a32b-4135-a38d-d0c3877758f3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c1a3db7-a32b-4135-a38d-d0c3877758f3",
"name": "6c1a3db7-a32b-4135-a38d-d0c3877758f3",
"status": "Failed",
"startTime": "2026-10-13T01:35:58.975421113Z",
"endTime": "2026-10-13T02:05:06.821915606Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-w6pg8-mtm5c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tk2gb5zvpbxs--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-w6pg8-mtm5c to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-w6pg8-mtm5c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tk2gb5zvpbxs--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-w6pg8-mtm5c to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8911104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c1a3db7-a32b-4135-a38d-d0c3877758f3
correlation ID: 3c00069f-3253-4f6a-bf00-774f9195172f
client request ID: 439e929c-f0c1-47b9-b779-b67e3bd14fff
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:139]: failed to create node pool
Expected success, but got an error:
<*fmt.wrapError | 0xd6f1ba92440>:
failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-tk2gb5zvpbxs" to finish creating: GET https://rp.j8911104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c1a3db7-a32b-4135-a38d-d0c3877758f3
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c1a3db7-a32b-4135-a38d-d0c3877758f3",
"name": "6c1a3db7-a32b-4135-a38d-d0c3877758f3",
"status": "Failed",
"startTime": "2026-10-13T01:35:58.975421113Z",
"endTime": "2026-10-13T02:05:06.821915606Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 1, want 2; node pool condition AllNodesHealthy is False: 1 of 2 machines are not healthy\nMachine ingress-cert-le-np-1-w6pg8-mtm5c: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tk2gb5zvpbxs--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-w6pg8-mtm5c to exist\n; node pool condition AllMachinesReady is False: 1 of 2 machines are not ready\nMachine ingress-cert-le-np-1-w6pg8-mtm5c: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-tk2gb5zvpbxs--managed/providers/Microsoft.Compute/virtualMachines/ingress-cert-le-np-1-w6pg8-mtm5c to exist\n"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j8911104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c1a3db7-a32b-4135-a38d-d0c3877758f3
correlation ID: 3c00069f-3253-4f6a-bf00-774f9195172f
client request ID: 439e929c-f0c1-47b9-b779-b67e3bd14fffalert (1)alert [hcp] KubeDeploymentReplicasMismatch fired Show raw failurealert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T02:25:28Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8911104-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.182:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8911104/providers/microsoft.monitor/accounts/hcps-j8911104", namespace="ocm-arohcpci01-2tdjlphid04r183f4uvlk38j32sjcgf8-v20260901", pod="arohcp-monitor-kube-state-metrics-7988dc648c-ctvv6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tdjlphid04r183f4uvlk38j32sjcgf8-v20260901/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-10-13T02:25:28Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8911104-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cloud-network-config-controller", endpoint="http", environment="ci01", instance="10.128.64.182:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8911104/providers/microsoft.monitor/accounts/hcps-j8911104", namespace="ocm-arohcpci01-2tdjlphid04r183f4uvlk38j32sjcgf8-v20260901", pod="arohcp-monitor-kube-state-metrics-7988dc648c-ctvv6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tdjlphid04r183f4uvlk38j32sjcgf8-v20260901/cloud-network-config-controller has not matched the expected number of replicas for longer than 30 minutes. |
|
| 6h 51m 28s | pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR |
westus3 | n/a | passed | #7439 (merged) | 0 | n/a | |
| 7h 14m 47s | pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR |
westus3 | e2e | failed | #7162 (merged) | 1 | e2e (1)cluster "<cluster>" did not complete the SRE-pinned version 4.21.36 Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.002s.
cluster "sre-version-pin-sjbwq2" did not complete the SRE-pinned version 4.21.36
Expected
<string>: 4.21.36 (Partial)
to equal
<string>: 4.21.36
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_version_pin.go:172]: Timed out after 2700.002s.
cluster "sre-version-pin-sjbwq2" did not complete the SRE-pinned version 4.21.36
Expected
<string>: 4.21.36 (Partial)
to equal
<string>: 4.21.36 |
|
| 7h 13m 29s | pull-ci-Azure-ARO-HCP-main-e2e-parallel | centralus | alert, e2e | failed | #6920 (open) | 3 | e2e (2)VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0x18ba92cf6f8>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x18ba9177500>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x18ba9176100>{
msg: "DNS for route host agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x18ba91760e0>{
msg: "DNS for agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:186]: failed to run simple web app and connectivity check app with cilium CNI
Unexpected error:
<*errors.joinError | 0x18ba92cf6f8>:
VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
{
errs: [
<*fmt.wrapError | 0x18ba9177500>{
msg: "VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x18ba9176100>{
msg: "DNS for route host agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <*fmt.wrapError | 0x18ba91760e0>{
msg: "DNS for agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-8jckw.apps.aro.cilium-cl.n1zs.j9547136.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded",
err: <context.deadlineExceededError>{},
},
},
},
],
}
occurredERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift Show raw failurefail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.42
Unexpected error:
<*fmt.wrapError | 0x1d502f6120>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-pkbxzb" in resourcegroup="rg-np-version-upgrade-pkbxzb-8r9ncw6xxd9f" to finish creating: GET https://rp.j9547136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d75dd276-e1d3-41db-8b80-3fc3ed007e5e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d75dd276-e1d3-41db-8b80-3fc3ed007e5e",
"name": "d75dd276-e1d3-41db-8b80-3fc3ed007e5e",
"status": "Failed",
"startTime": "2026-10-13T01:17:18.97912251Z",
"endTime": "2026-10-13T01:46:20.604608943Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9547136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d75dd276-e1d3-41db-8b80-3fc3ed007e5e
correlation ID: 4e3d5b66-dbd0-45e8-931a-5c34e6be67a5
client request ID: 33b8ec2f-ff38-476b-ac0b-f1b2c13ca519
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.42
Unexpected error:
<*fmt.wrapError | 0x1d502f6120>:
failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-pkbxzb" in resourcegroup="rg-np-version-upgrade-pkbxzb-8r9ncw6xxd9f" to finish creating: GET https://rp.j9547136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d75dd276-e1d3-41db-8b80-3fc3ed007e5e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: DeadlineExceeded
--------------------------------------------------------------------------------
{
"id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d75dd276-e1d3-41db-8b80-3fc3ed007e5e",
"name": "d75dd276-e1d3-41db-8b80-3fc3ed007e5e",
"status": "Failed",
"startTime": "2026-10-13T01:17:18.97912251Z",
"endTime": "2026-10-13T01:46:20.604608943Z",
"error": {
"code": "DeadlineExceeded",
"message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet"
}
}
--------------------------------------------------------------------------------
<Azure error internals omitted>
Azure request IDs:
GET https://rp.j9547136.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d75dd276-e1d3-41db-8b80-3fc3ed007e5e
correlation ID: 4e3d5b66-dbd0-45e8-931a-5c34e6be67a5
client request ID: 33b8ec2f-ff38-476b-ac0b-f1b2c13ca519
occurredalert (1)alert [svc] LeaderElectionLeaseStale fired Show raw failurealert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-13T02:15:50Z Severity: Sev3 Labels: alertname="LeaderElectionLeaseStale", cluster="ci01-j9547136-mgmt-1", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.164:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9547136/providers/microsoft.monitor/accounts/services-j9547136", namespace="klusterlet-2tdj9mm19ano4hsp1ejpct9vbiqi8t8u", pod="arohcp-monitor-kube-state-metrics-7988dc648c-448ch", prometheus="prometheus/prometheus", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Leader election lease work-manager in namespace klusterlet-2tdj9mm19ano4hsp1ejpct9vbiqi8t8u on cluster ci01-j9547136-mgmt-1 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running. |