CIHealth Run Log

Environment: DEV

Time (UTC)	Runtime	Job	Region	Failed at	Result	PR	Failed tests	Details
23:07:44	2h 18m 42s	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head a38be30 · merge 85f478e	canadacentral	e2e	failed	⚠#7739 (merged)	1	e2e (1) ERROR CODE: NotFound; detail message The resource record '.e2e-dns-rdkvkj' does not exist in resource group '<resource-group>' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'.; provider Microsoft.Network 23:07:44 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-4-22-cprxqg" Unexpected error: <errors.joinError \| 0x30c853d502b8>: VerifyIngressDNSRecord(aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud) timed out after 10m0.003s: DNS A record ".e2e-dns-rdkvkj" not yet present in zone rg-candidate-4-22-hzrgxmfgtllm-candidate-cprxqg--managed/aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-hzrgxmfgtllm-candidate-cprxqg--managed/providers/Microsoft.Network/dnsZones/aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud/A/.e2e-dns-rdkvkj -------------------------------------------------------------------------------- RESPONSE 404: 404 Not Found ERROR CODE: NotFound -------------------------------------------------------------------------------- { "code": "NotFound", "message": "The resource record '.e2e-dns-rdkvkj' does not exist in resource group 'rg-candidate-4-22-hzrgxmfgtllm-candidate-cprxqg--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'." } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-hzrgxmfgtllm-candidate-cprxqg--managed/providers/Microsoft.Network/dnsZones/aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud/A/.e2e-dns-rdkvkj correlation ID: 984d9387-98d9-461e-8f5b-657273ddef57 client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:246]: failed to verify cluster operators on cluster "cluster-candidate-4-22-cprxqg" Unexpected error: <errors.joinError \| 0x30c853d502b8>: VerifyIngressDNSRecord(aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud) failed: VerifyIngressDNSRecord(aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud) timed out after 10m0.003s: DNS A record ".e2e-dns-rdkvkj" not yet present in zone rg-candidate-4-22-hzrgxmfgtllm-candidate-cprxqg--managed/aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-hzrgxmfgtllm-candidate-cprxqg--managed/providers/Microsoft.Network/dnsZones/aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud/A/.e2e-dns-rdkvkj -------------------------------------------------------------------------------- RESPONSE 404: 404 Not Found ERROR CODE: NotFound -------------------------------------------------------------------------------- { "code": "NotFound", "message": "The resource record '.e2e-dns-rdkvkj' does not exist in resource group 'rg-candidate-4-22-hzrgxmfgtllm-candidate-cprxqg--managed' of subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'." } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-hzrgxmfgtllm-candidate-cprxqg--managed/providers/Microsoft.Network/dnsZones/aro.n9p8g3s6f9q9f8h.s668.j0574848.hcp.osadev.cloud/A/*.e2e-dns-rdkvkj correlation ID: 984d9387-98d9-461e-8f5b-657273ddef57 client request ID: unavailable occurred
17:30:28	2h 21m 35s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 8a24997	canadacentral	alert, e2e	failed	⚠#7712 (closed)	4	e2e (2) DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded 17:30:28 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20 Unexpected error: <fmt.wrapError \| 0xd75a95e43e0>: DNS for route host agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded { msg: "DNS for route host agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded", err: <fmt.wrapError \| 0xd75a8fba020>{ msg: "DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded", err: <context.deadlineExceededError>{}, }, } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20 Unexpected error: <fmt.wrapError \| 0xd75a95e43e0>: DNS for route host agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded { msg: "DNS for route host agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded", err: <fmt.wrapError \| 0xd75a8fba020>{ msg: "DNS for agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-qx6cd.apps.aro.s4p6p0t9k4o2s8h.fx2j.j3671936.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded", err: <context.deadlineExceededError>{}, }, } occurred missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14240, acm-agent=0 17:30:28 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 630.763s. Expected success, but got an error: <errors.errorString \| 0x2d6e687aea00>: missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14240, acm-agent=0 { s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14240, acm-agent=0", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 630.763s. Expected success, but got an error: <errors.errorString \| 0x2d6e687aea00>: missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14240, acm-agent=0 { s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14240, acm-agent=0", } alert (2) alert [svc] KubeDeploymentReplicasMismatch fired 17:30:28 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 3 time(s) Firing 1: State: Resolved Started: 2026-10-25T19:03:55Z Ended: 2026-10-25T19:10:56Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-7vdl7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-25T19:03:55Z Ended: 2026-10-25T19:09:56Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-7vdl7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. Firing 3: State: Resolved Started: 2026-10-25T19:03:55Z Ended: 2026-10-25T19:10:56Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-7vdl7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2tlvp310lq0lqg3o9bn0eeqgoijvib5c/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes. alert [svc] KubePodCrashLooping fired 17:30:28 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubePodCrashLooping does not fire · suite aro-hcp-tests Show raw failure alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-25T18:47:54Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-bfwrz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="97b76fc8-3f29-44d9-83c4-855c4cca2cf8" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-bfwrz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-25T18:47:54Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3671936-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.215:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-mftk6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c4f6f9ce-1370-407b-a328-72082e9577a0" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-mftk6 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-25T18:50:58Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3671936-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.215:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-mftk6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="c4f6f9ce-1370-407b-a328-72082e9577a0" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-mftk6 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-25T18:50:58Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3671936-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.162:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3671936/providers/microsoft.monitor/accounts/services-j3671936", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-bfwrz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="97b76fc8-3f29-44d9-83c4-855c4cca2cf8" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-bfwrz (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
17:30:16	22m 13s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0b660a1	n/a	unknown	failed	#6191 (open)	1	non-artifact-backed failure (no junit artifacts)
16:26:43	2h 5m 2s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 8cc8fdb	westus3	n/a	passed	#6364 (open)	0	n/a
13:30:21	2h 14m 11s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 322d479 · merge 322d479	centralus	n/a	passed	#7597 (merged)	0	n/a
12:30:14	2h 23m 53s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 8a24997	centralus	alert, e2e	failed	⚠#7712 (closed)	2	e2e (1) missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14128, acm-agent=0 12:30:14 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 620.639s. Expected success, but got an error: <errors.errorString \| 0x1cfcad623430>: missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14128, acm-agent=0 { s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14128, acm-agent=0", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 620.639s. Expected success, but got an error: <errors.errorString \| 0x1cfcad623430>: missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14128, acm-agent=0 { s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=14128, acm-agent=0", } alert (1) alert [svc] KubePodCrashLooping fired 12:30:14 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubePodCrashLooping does not fire · suite aro-hcp-tests Show raw failure alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-25T13:52:18Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j8072320-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8072320/providers/microsoft.monitor/accounts/services-j8072320", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-jdq5f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9c6d2039-7988-4be2-8890-d290ed7d2d20" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-jdq5f (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-25T13:54:18Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j8072320-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8072320/providers/microsoft.monitor/accounts/services-j8072320", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-jdq5f", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="9c6d2039-7988-4be2-8890-d290ed7d2d20" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-jdq5f (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-25T13:55:20Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j8072320-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.141:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8072320/providers/microsoft.monitor/accounts/services-j8072320", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-h6zb9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8720bf8b-027c-410b-9181-eafea5b14401" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-h6zb9 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-25T13:55:20Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j8072320-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.141:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8072320/providers/microsoft.monitor/accounts/services-j8072320", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-h6zb9", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="8720bf8b-027c-410b-9181-eafea5b14401" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-h6zb9 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
12:30:03	2h 20m 40s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0b660a1	centralus	alert, e2e	failed	#6191 (open)	5	e2e (2) certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist 12:30:03 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:510]: Timed out after 600.003s. Certificate should be issued by Let's Encrypt staging via ACME DNS-01 Expected success, but got an error: <errors.errorString \| 0x25b4e2d279a0>: certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist { s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist", } fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:510]: Timed out after 600.003s. Certificate should be issued by Let's Encrypt staging via ACME DNS-01 Expected success, but got an error: <errors.errorString \| 0x25b4e2d279a0>: certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist { s: "certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist", } ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 12:30:03 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0x209528ec2340>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ktmjg5k5mr4p" to finish creating: GET https://rp.j2101760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564", "name": "cd12106f-dc74-4bb2-88d9-552815761564", "status": "Failed", "startTime": "2026-10-25T13:33:58.83934802Z", "endTime": "2026-10-25T14:03:01.452063547Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j2101760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564 correlation ID: a999ca91-1f04-4667-8416-be09ce537e26 client request ID: e9d482d5-6b2f-4cbd-99d8-1b5bb0124d29 occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0x209528ec2340>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-ktmjg5k5mr4p" to finish creating: GET https://rp.j2101760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564", "name": "cd12106f-dc74-4bb2-88d9-552815761564", "status": "Failed", "startTime": "2026-10-25T13:33:58.83934802Z", "endTime": "2026-10-25T14:03:01.452063547Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j2101760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cd12106f-dc74-4bb2-88d9-552815761564 correlation ID: a999ca91-1f04-4667-8416-be09ce537e26 client request ID: e9d482d5-6b2f-4cbd-99d8-1b5bb0124d29 occurred alert (3) alert [hcp] KubePodCrashLooping fired 12:30:03 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-25T13:57:52Z Ended: 2026-10-25T14:23:56Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="b19b2a4f-abec-4efb-8fa1-a0f49de6f865" Description: Pod ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-25T13:57:52Z Ended: 2026-10-25T14:23:56Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="b19b2a4f-abec-4efb-8fa1-a0f49de6f865" Description: Pod ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff"). alert [hcp] KubeDeploymentReplicasMismatch fired 12:30:03 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-25T14:09:53Z Ended: 2026-10-25T14:23:58Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xj2t5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-10-25T14:10:54Z Ended: 2026-10-25T14:19:54Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xj2t5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert [hcp] KubeStatefulSetReplicasMismatch fired 12:30:03 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-10-25T13:55:59Z Ended: 2026-10-25T14:19:54Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xj2t5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-10-25T14:06:53Z Ended: 2026-10-25T14:19:54Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2101760-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.228:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2101760/providers/microsoft.monitor/accounts/hcps-j2101760", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901", pod="arohcp-monitor-kube-state-metrics-85d48f46ff-xj2t5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2tlrehr6k1m1drteiq3dbt6soul2kbq3-v20260901/etcd has not matched the expected number of replicas for longer than 15 minutes.
11:17:06	1h 2m 33s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 322d479 · merge 322d479	canadacentral	provision	failed	★#7597 (merged)	1	provision (1) failed to run istio upgrade step: failed to complete canary: complete-canary: ARM operation timed out or failed: context deadline exceeded 11:17:06 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade · suite templatize-pipeline Show raw failure time=2026-10-25T11:53:02.786Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade description="Step istio-upgrade\n Kind: IstioUpgrade\n AKSCluster: {{ svc.aks.name }}\n" time=2026-10-25T11:53:04.190Z level=INFO msg="Istio upgrade -- cluster state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade k8sVersion=1.35 provisioningState=Succeeded installedRevisions=[asm-1-29] availableUpgrades=[asm-1-30] upgradeInProgress=false time=2026-10-25T11:53:04.190Z level=INFO msg="Upgrading to svc.istio.versions target" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade from=asm-1-29 to=asm-1-30 k8sVersion=1.35 time=2026-10-25T11:53:05.198Z level=INFO msg="Istio upgrade -- mesh state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade namespaces=0 controlPlanes=asm-1-29(2/2) ingressGateways=1 webhooks=1 configMaps=2 time=2026-10-25T11:53:05.198Z level=INFO msg="Starting canary -- installing target alongside current" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade time=2026-10-25T11:53:05.198Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary cluster=ci01-j8850688-svc time=2026-10-25T11:56:09.852Z level=INFO msg="Cluster provisioning state" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade state=Succeeded time=2026-10-25T11:56:09.852Z level=INFO msg="ARM operation complete" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=start-canary elapsed=3m5s time=2026-10-25T11:56:09.852Z level=INFO msg="Step 1/9: Ensuring MISE ext-authz ConfigMap" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade time=2026-10-25T11:56:10.416Z level=INFO msg="ConfigMap created" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade/revision-configmap name=istio-shared-configmap-asm-1-30 time=2026-10-25T11:56:10.416Z level=INFO msg="Step 2/9: Checking tag-based namespace safety" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade time=2026-10-25T11:56:10.416Z level=INFO msg="Step 3/9: Verifying control plane health and flipping tag webhook" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade time=2026-10-25T11:56:11.207Z level=INFO msg="Created revision tag webhook" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade/revision-tag webhook=istio-revision-tag-prod-stable-aks-istio-system revision=asm-1-30 time=2026-10-25T11:56:11.207Z level=INFO msg="Step 4/9: Ensuring ingress gateway annotations" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade time=2026-10-25T11:56:11.237Z level=INFO msg="Applying annotations" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade/ingress-annotations service=aks-istio-ingressgateway-external time=2026-10-25T11:56:11.450Z level=INFO msg="Step 5/9: Migrating workloads to target revision" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade time=2026-10-25T11:56:11.510Z level=INFO msg="Step 6/9: Running health check" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade time=2026-10-25T11:56:11.971Z level=INFO msg="Health check passed -- checking for orphaned workloads before completing canary" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade time=2026-10-25T11:56:11.971Z level=INFO msg="Step 7/9: Running orphan guard" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade time=2026-10-25T11:56:11.971Z level=INFO msg="No orphaned workloads -- completing canary" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade time=2026-10-25T11:56:11.971Z level=INFO msg="Step 8/9: Completing canary via ARM (point of no return)" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade cluster=ci01-j8850688-svc versions=asm-1-30 logger=istio-upgrade time=2026-10-25T11:56:11.971Z level=INFO msg="Starting ARM operation" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade stepID=Microsoft.Azure.ARO.HCP.Service.Infra/service/istio-upgrade operation=complete-canary cluster=ci01-j8850688-svc time=2026-10-25T12:11:14.972Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=istio-upgrade err="failed to run istio upgrade step: failed to complete canary: complete-canary: ARM operation timed out or failed: context deadline exceeded"
09:55:02	2h 29m 14s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 860fe9f	centralus	alert, e2e	failed	⚠#6081 (open)	2	e2e (1) ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet; provider Microsoft.RedHatOpenShift 09:55:02 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster Unexpected error: <fmt.wrapError \| 0x34e2e46f8680>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-mfmrpz5xbrdj" to finish creating: GET https://rp.j1877504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14", "name": "3d1a2520-89cf-4c42-a91c-ec725b89af14", "status": "Failed", "startTime": "2026-10-25T11:04:09.039925979Z", "endTime": "2026-10-25T11:23:09.310567948Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j1877504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14 correlation ID: 9acdbad6-f09a-46a0-a926-c738c62f3c07 client request ID: 916943ff-0406-46da-9830-2889e6355ff3 occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:178]: failed to create node pool np-1 for pull secret cluster Unexpected error: <fmt.wrapError \| 0x34e2e46f8680>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-mfmrpz5xbrdj" to finish creating: GET https://rp.j1877504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14", "name": "3d1a2520-89cf-4c42-a91c-ec725b89af14", "status": "Failed", "startTime": "2026-10-25T11:04:09.039925979Z", "endTime": "2026-10-25T11:23:09.310567948Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift node pool not cached yet" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j1877504.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3d1a2520-89cf-4c42-a91c-ec725b89af14 correlation ID: 9acdbad6-f09a-46a0-a926-c738c62f3c07 client request ID: 916943ff-0406-46da-9830-2889e6355ff3 occurred alert (1) alert [svc] ClusterCredentialMetricMissing fired 09:55:02 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ClusterCredentialMetricMissing does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-25T11:11:08Z Severity: Sev3 Labels: alertname="ClusterCredentialMetricMissing", cluster="ci01-j1877504-svc", severity="warning" Description: Credential refresher expiration metric is missing for service cluster ci01-j1877504-svc.
09:54:40	2h 31m 1s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0b660a1	westus3	alert, e2e	failed	⚠#6191 (open)	22	e2e (18) ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0x105dbf764180>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-mv5sq4l7jv4r" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/35ffd4e9-2ccb-4812-b71c-6d6a35fe8b0b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/35ffd4e9-2ccb-4812-b71c-6d6a35fe8b0b", "name": "35ffd4e9-2ccb-4812-b71c-6d6a35fe8b0b", "status": "Failed", "startTime": "2026-10-25T11:01:21.658859222Z", "endTime": "2026-10-25T11:30:25.924003842Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: capi-provider; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/35ffd4e9-2ccb-4812-b71c-6d6a35fe8b0b correlation ID: d887bb88-655e-4cf4-84f8-c59ec1d133f4 client request ID: 066a4e25-cc0e-4d6f-a9c5-8e71913c3b6c occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:114]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0x105dbf764180>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-mv5sq4l7jv4r" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/35ffd4e9-2ccb-4812-b71c-6d6a35fe8b0b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/35ffd4e9-2ccb-4812-b71c-6d6a35fe8b0b", "name": "35ffd4e9-2ccb-4812-b71c-6d6a35fe8b0b", "status": "Failed", "startTime": "2026-10-25T11:01:21.658859222Z", "endTime": "2026-10-25T11:30:25.924003842Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: capi-provider; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/35ffd4e9-2ccb-4812-b71c-6d6a35fe8b0b correlation ID: d887bb88-655e-4cf4-84f8-c59ec1d133f4 client request ID: 066a4e25-cc0e-4d6f-a9c5-8e71913c3b6c occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [kube-controller-manager]; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0x3a8a07a1e540>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-rvs4gzxj9lgg" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63403578-fb62-456b-b40e-41ef79e2e399 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63403578-fb62-456b-b40e-41ef79e2e399", "name": "63403578-fb62-456b-b40e-41ef79e2e399", "status": "Failed", "startTime": "2026-10-25T11:02:39.014043295Z", "endTime": "2026-10-25T11:31:45.790547256Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m13s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63403578-fb62-456b-b40e-41ef79e2e399 correlation ID: 38478f64-7ba7-40aa-b720-a7f94d356d8b client request ID: 82836f4c-341e-4d1e-88f4-c55b4e681a30 occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0x3a8a07a1e540>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-rvs4gzxj9lgg" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63403578-fb62-456b-b40e-41ef79e2e399 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63403578-fb62-456b-b40e-41ef79e2e399", "name": "63403578-fb62-456b-b40e-41ef79e2e399", "status": "Failed", "startTime": "2026-10-25T11:02:39.014043295Z", "endTime": "2026-10-25T11:31:45.790547256Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m13s ago; hosted cluster degraded: UnavailableReplicas: kube-controller-manager deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63403578-fb62-456b-b40e-41ef79e2e399 correlation ID: 38478f64-7ba7-40aa-b720-a7f94d356d8b client request ID: 82836f4c-341e-4d1e-88f4-c55b4e681a30 occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0x3f211b784120>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-fqjfphzpbt4x" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc5b2708-f9a2-4d86-915e-eb8ba91cbd59 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc5b2708-f9a2-4d86-915e-eb8ba91cbd59", "name": "dc5b2708-f9a2-4d86-915e-eb8ba91cbd59", "status": "Failed", "startTime": "2026-10-25T11:01:22.163428437Z", "endTime": "2026-10-25T11:30:25.784218117Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.19 is Partial (want Completed), started 19m44s ago; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc5b2708-f9a2-4d86-915e-eb8ba91cbd59 correlation ID: 16b92b93-ef61-4f4d-9bbb-a63f0c7ebf4f client request ID: cc5cc315-91ce-4032-81e3-c0e49c80e860 occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:101]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0x3f211b784120>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-fqjfphzpbt4x" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc5b2708-f9a2-4d86-915e-eb8ba91cbd59 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc5b2708-f9a2-4d86-915e-eb8ba91cbd59", "name": "dc5b2708-f9a2-4d86-915e-eb8ba91cbd59", "status": "Failed", "startTime": "2026-10-25T11:01:22.163428437Z", "endTime": "2026-10-25T11:30:25.784218117Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.19 is Partial (want Completed), started 19m44s ago; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/dc5b2708-f9a2-4d86-915e-eb8ba91cbd59 correlation ID: 16b92b93-ef61-4f4d-9bbb-a63f0c7ebf4f client request ID: cc5cc315-91ce-4032-81e3-c0e49c80e860 occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0x134cca4de620>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-smqlw28gpmfk" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d4d8ca67-31e0-4a7f-b038-d6a3a610c1f2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d4d8ca67-31e0-4a7f-b038-d6a3a610c1f2", "name": "d4d8ca67-31e0-4a7f-b038-d6a3a610c1f2", "status": "Failed", "startTime": "2026-10-25T11:01:59.115997025Z", "endTime": "2026-10-25T11:31:07.16366343Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m52s ago" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d4d8ca67-31e0-4a7f-b038-d6a3a610c1f2 correlation ID: 9f778ec9-bb64-4e1c-9a2f-a32317e4bb4a client request ID: bfadd86d-0c2c-4751-a28f-5658e7a124e6 occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:98]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0x134cca4de620>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-smqlw28gpmfk" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d4d8ca67-31e0-4a7f-b038-d6a3a610c1f2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d4d8ca67-31e0-4a7f-b038-d6a3a610c1f2", "name": "d4d8ca67-31e0-4a7f-b038-d6a3a610c1f2", "status": "Failed", "startTime": "2026-10-25T11:01:59.115997025Z", "endTime": "2026-10-25T11:31:07.16366343Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m52s ago" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d4d8ca67-31e0-4a7f-b038-d6a3a610c1f2 correlation ID: 9f778ec9-bb64-4e1c-9a2f-a32317e4bb4a client request ID: bfadd86d-0c2c-4751-a28f-5658e7a124e6 occurred ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url> 09:54:40 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:82]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0x2401cda17700>: failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-np-labels-taints-6lgfpz" in resourcegroup="rg-np-labels-taints-2pq6dbldqrjk" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-2pq6dbldqrjk/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-6lgfpz/operationStatuses/08584086797384633997 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: f50cf33f-8bc4-4ce0-8a7c-d816cfc4e960", "details": [ { "code": "Conflict", "message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-bvffjvcy6qo3o' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-2pq6dbldqrjk/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-6lgfpz/operationStatuses/08584086797384633997 correlation ID: 9ef1eaf3-8ba8-425e-8fe5-70f8311af59e client request ID: unavailable occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:82]: failed to create cluster customer resources Unexpected error: <fmt.wrapError \| 0x2401cda17700>: failed to create customer-infra: failed waiting for deployment "customer-infra-cluster-np-labels-taints-6lgfpz" in resourcegroup="rg-np-labels-taints-2pq6dbldqrjk" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-2pq6dbldqrjk/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-6lgfpz/operationStatuses/08584086797384633997 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details. Correlation ID: f50cf33f-8bc4-4ce0-8a7c-d816cfc4e960", "details": [ { "code": "Conflict", "message": "{\r\n \"error\": {\r\n \"code\": \"VaultAlreadyExists\",\r\n \"message\": \"The vault name 'cust-kv-bvffjvcy6qo3o' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-np-labels-taints-2pq6dbldqrjk/providers/Microsoft.Resources/deployments/customer-infra-cluster-np-labels-taints-6lgfpz/operationStatuses/08584086797384633997 correlation ID: 9ef1eaf3-8ba8-425e-8fe5-70f8311af59e client request ID: unavailable occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [capi-provider, catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-zgspqd" with version 4.22.18 on candidate channel Unexpected error: <fmt.wrapError \| 0x1120db708700>: failed to create HCP cluster cluster-zstream-4-22-zgspqd: failed waiting for cluster="cluster-zstream-4-22-zgspqd" in resourcegroup="rg-zstream-upgrade-4-22-4qv6w5xvr977" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffe95342-e02c-4496-883e-5b1bf611d99e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffe95342-e02c-4496-883e-5b1bf611d99e", "name": "ffe95342-e02c-4496-883e-5b1bf611d99e", "status": "Failed", "startTime": "2026-10-25T11:01:19.965849634Z", "endTime": "2026-10-25T11:30:25.598823992Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, olm-operator, capi-provider, cluster-network-operator, cluster-node-tuning-operator, ingress-operator, dns-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, packageserver, redhat-operators-catalog, openshift-route-controller-manager, community-operators-catalog, certified-operators-catalog, cluster-storage-operator, catalog-operator, openshift-controller-manager, cluster-version-operator, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffe95342-e02c-4496-883e-5b1bf611d99e correlation ID: e700165f-ec9f-48e4-abf2-18b14bbd9d95 client request ID: cdbafbf0-d5e0-4ed7-8b28-b50e4bf8f6b8 occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-22-zgspqd" with version 4.22.18 on candidate channel Unexpected error: <fmt.wrapError \| 0x1120db708700>: failed to create HCP cluster cluster-zstream-4-22-zgspqd: failed waiting for cluster="cluster-zstream-4-22-zgspqd" in resourcegroup="rg-zstream-upgrade-4-22-4qv6w5xvr977" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffe95342-e02c-4496-883e-5b1bf611d99e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffe95342-e02c-4496-883e-5b1bf611d99e", "name": "ffe95342-e02c-4496-883e-5b1bf611d99e", "status": "Failed", "startTime": "2026-10-25T11:01:19.965849634Z", "endTime": "2026-10-25T11:30:25.598823992Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator, olm-operator, capi-provider, cluster-network-operator, cluster-node-tuning-operator, ingress-operator, dns-operator, csi-snapshot-controller-operator, cluster-image-registry-operator, packageserver, redhat-operators-catalog, openshift-route-controller-manager, community-operators-catalog, certified-operators-catalog, cluster-storage-operator, catalog-operator, openshift-controller-manager, cluster-version-operator, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [capi-provider deployment has 1 unavailable replicas, catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ffe95342-e02c-4496-883e-5b1bf611d99e correlation ID: e700165f-ec9f-48e4-abf2-18b14bbd9d95 client request ID: cdbafbf0-d5e0-4ed7-8b28-b50e4bf8f6b8 occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test SRE should return 409 when boot diagnostics is disabled on a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error: <errors.joinError \| 0x381116f3a0c0>: resource group "admin-api-serialconsole-bootdiag-l9k6ftz85vvb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-l9k6ftz85vvb" to finish deleting: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/894117ff-70c7-42e1-b1b8-b8df80eb0d8c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/894117ff-70c7-42e1-b1b8-b8df80eb0d8c", "name": "894117ff-70c7-42e1-b1b8-b8df80eb0d8c", "status": "Failed", "startTime": "2026-10-25T11:28:30.111139221Z", "endTime": "2026-10-25T11:52:32.090609688Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/894117ff-70c7-42e1-b1b8-b8df80eb0d8c correlation ID: d229c55a-e375-4f40-bb2a-f3ad97610736 client request ID: aa4bc9f4-d5d8-494d-94b7-252189caf6ce occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error: <errors.joinError \| 0x381116f3a0c0>: resource group "admin-api-serialconsole-bootdiag-l9k6ftz85vvb": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-l9k6ftz85vvb" to finish deleting: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/894117ff-70c7-42e1-b1b8-b8df80eb0d8c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/894117ff-70c7-42e1-b1b8-b8df80eb0d8c", "name": "894117ff-70c7-42e1-b1b8-b8df80eb0d8c", "status": "Failed", "startTime": "2026-10-25T11:28:30.111139221Z", "endTime": "2026-10-25T11:52:32.090609688Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/894117ff-70c7-42e1-b1b8-b8df80eb0d8c correlation ID: d229c55a-e375-4f40-bb2a-f3ad97610736 client request ID: aa4bc9f4-d5d8-494d-94b7-252189caf6ce occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator]; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-5-0-vfvmsnwsd5sg/cluster-candidate-5-0-lh7h4v should provision Unexpected error: <fmt.wrapError \| 0x274288768300>: failed to create HCP cluster cluster-candidate-5-0-lh7h4v: failed waiting for cluster="cluster-candidate-5-0-lh7h4v" in resourcegroup="rg-candidate-5-0-vfvmsnwsd5sg" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8664d31-ed00-44b7-89f1-f9abec4ea6b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8664d31-ed00-44b7-89f1-f9abec4ea6b8", "name": "f8664d31-ed00-44b7-89f1-f9abec4ea6b8", "status": "Failed", "startTime": "2026-10-25T11:01:22.79701272Z", "endTime": "2026-10-25T11:30:25.820223913Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.7 is Partial (want Completed), started 20m56s ago; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8664d31-ed00-44b7-89f1-f9abec4ea6b8 correlation ID: d572d016-0277-4113-8a96-1da300eb89da client request ID: a981c6e6-813e-4373-81f7-4aab2cf9a886 occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-5-0-vfvmsnwsd5sg/cluster-candidate-5-0-lh7h4v should provision Unexpected error: <fmt.wrapError \| 0x274288768300>: failed to create HCP cluster cluster-candidate-5-0-lh7h4v: failed waiting for cluster="cluster-candidate-5-0-lh7h4v" in resourcegroup="rg-candidate-5-0-vfvmsnwsd5sg" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8664d31-ed00-44b7-89f1-f9abec4ea6b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8664d31-ed00-44b7-89f1-f9abec4ea6b8", "name": "f8664d31-ed00-44b7-89f1-f9abec4ea6b8", "status": "Failed", "startTime": "2026-10-25T11:01:22.79701272Z", "endTime": "2026-10-25T11:30:25.820223913Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.7 is Partial (want Completed), started 20m56s ago; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f8664d31-ed00-44b7-89f1-f9abec4ea6b8 correlation ID: d572d016-0277-4113-8a96-1da300eb89da client request ID: a981c6e6-813e-4373-81f7-4aab2cf9a886 occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0x304be2e14ae0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-xps9m7pqkcln" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621e8150-158d-427d-a7d5-68a63e6ec477 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621e8150-158d-427d-a7d5-68a63e6ec477", "name": "621e8150-158d-427d-a7d5-68a63e6ec477", "status": "Failed", "startTime": "2026-10-25T11:02:05.420485549Z", "endTime": "2026-10-25T11:31:06.91227051Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m0s ago" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621e8150-158d-427d-a7d5-68a63e6ec477 correlation ID: 38706e55-d1d9-4140-8dbb-06ce2b05e393 client request ID: 8c26f98a-a05c-479f-9514-5fa1e0ca4e1e occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0x304be2e14ae0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-xps9m7pqkcln" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621e8150-158d-427d-a7d5-68a63e6ec477 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621e8150-158d-427d-a7d5-68a63e6ec477", "name": "621e8150-158d-427d-a7d5-68a63e6ec477", "status": "Failed", "startTime": "2026-10-25T11:02:05.420485549Z", "endTime": "2026-10-25T11:31:06.91227051Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m0s ago" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/621e8150-158d-427d-a7d5-68a63e6ec477 correlation ID: 38706e55-d1d9-4140-8dbb-06ce2b05e393 client request ID: 8c26f98a-a05c-479f-9514-5fa1e0ca4e1e occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [hosted-cluster-config-operator]; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test Customer should be able to retrieve cluster and nodepool status active versions · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:80]: failed to create HCP cluster "v20261001" via v20261001 Unexpected error: <fmt.wrapError \| 0x2a59ed8de2a0>: failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-xf7h6gdhg87b" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c13cc8b-efe6-4584-9ae2-571f7f85e94b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c13cc8b-efe6-4584-9ae2-571f7f85e94b", "name": "0c13cc8b-efe6-4584-9ae2-571f7f85e94b", "status": "Failed", "startTime": "2026-10-25T11:01:20.835629052Z", "endTime": "2026-10-25T11:30:25.907970891Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c13cc8b-efe6-4584-9ae2-571f7f85e94b correlation ID: f5a00756-4d2f-4428-b3ea-e6cf5e3ef0a1 client request ID: 03254b25-a5c1-4e3b-88dd-5aeb8bec557c occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:80]: failed to create HCP cluster "v20261001" via v20261001 Unexpected error: <fmt.wrapError \| 0x2a59ed8de2a0>: failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001-xf7h6gdhg87b" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c13cc8b-efe6-4584-9ae2-571f7f85e94b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c13cc8b-efe6-4584-9ae2-571f7f85e94b", "name": "0c13cc8b-efe6-4584-9ae2-571f7f85e94b", "status": "Failed", "startTime": "2026-10-25T11:01:20.835629052Z", "endTime": "2026-10-25T11:30:25.907970891Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: hosted-cluster-config-operator; hosted cluster degraded: UnavailableReplicas: hosted-cluster-config-operator deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0c13cc8b-efe6-4584-9ae2-571f7f85e94b correlation ID: f5a00756-4d2f-4428-b3ea-e6cf5e3ef0a1 client request ID: 03254b25-a5c1-4e3b-88dd-5aeb8bec557c occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [resourceTeardown] tearing down namespaces; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error: <errors.joinError \| 0x22bced7141e0>: resource group "rg-cluster-nsg-subnet-reuse-xcl2892ldqng": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-xcl2892ldqng" to finish deleting: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/954cc982-3e76-4b31-9df6-fcc83a76d91a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/954cc982-3e76-4b31-9df6-fcc83a76d91a", "name": "954cc982-3e76-4b31-9df6-fcc83a76d91a", "status": "Failed", "startTime": "2026-10-25T11:21:43.829823922Z", "endTime": "2026-10-25T11:45:52.860717396Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/954cc982-3e76-4b31-9df6-fcc83a76d91a correlation ID: dc5c263c-66f3-4b74-a013-b67fbdf80f9a client request ID: 3eb7aaab-ba2a-414a-b973-c3e5012b0c78 occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:310]: Unexpected error: <errors.joinError \| 0x22bced7141e0>: resource group "rg-cluster-nsg-subnet-reuse-xcl2892ldqng": failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-xcl2892ldqng" to finish deleting: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/954cc982-3e76-4b31-9df6-fcc83a76d91a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/954cc982-3e76-4b31-9df6-fcc83a76d91a", "name": "954cc982-3e76-4b31-9df6-fcc83a76d91a", "status": "Failed", "startTime": "2026-10-25T11:21:43.829823922Z", "endTime": "2026-10-25T11:45:52.860717396Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/954cc982-3e76-4b31-9df6-fcc83a76d91a correlation ID: dc5c263c-66f3-4b74-a013-b67fbdf80f9a client request ID: 3eb7aaab-ba2a-414a-b973-c3e5012b0c78 occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.23 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-4-23-8bzfjqhxqbsv/cluster-candidate-4-23-p8j4lz should provision Unexpected error: <fmt.wrapError \| 0x222687d52140>: failed to create HCP cluster cluster-candidate-4-23-p8j4lz: failed waiting for cluster="cluster-candidate-4-23-p8j4lz" in resourcegroup="rg-candidate-4-23-8bzfjqhxqbsv" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b6dd64ca-a8ee-4bd8-a4a0-a7a645bb0fc6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b6dd64ca-a8ee-4bd8-a4a0-a7a645bb0fc6", "name": "b6dd64ca-a8ee-4bd8-a4a0-a7a645bb0fc6", "status": "Failed", "startTime": "2026-10-25T11:01:20.644379442Z", "endTime": "2026-10-25T11:30:25.911986077Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.23.0-ec.1 is Partial (want Completed), started 21m2s ago" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b6dd64ca-a8ee-4bd8-a4a0-a7a645bb0fc6 correlation ID: 6870aa52-f7db-4c55-a33b-9785295a817d client request ID: 3eeba58c-cd03-4b39-8109-cd7d06f1728d occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:126]: HCP cluster rg-candidate-4-23-8bzfjqhxqbsv/cluster-candidate-4-23-p8j4lz should provision Unexpected error: <fmt.wrapError \| 0x222687d52140>: failed to create HCP cluster cluster-candidate-4-23-p8j4lz: failed waiting for cluster="cluster-candidate-4-23-p8j4lz" in resourcegroup="rg-candidate-4-23-8bzfjqhxqbsv" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b6dd64ca-a8ee-4bd8-a4a0-a7a645bb0fc6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b6dd64ca-a8ee-4bd8-a4a0-a7a645bb0fc6", "name": "b6dd64ca-a8ee-4bd8-a4a0-a7a645bb0fc6", "status": "Failed", "startTime": "2026-10-25T11:01:20.644379442Z", "endTime": "2026-10-25T11:30:25.911986077Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.23.0-ec.1 is Partial (want Completed), started 21m2s ago" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b6dd64ca-a8ee-4bd8-a4a0-a7a645bb0fc6 correlation ID: 6870aa52-f7db-4c55-a33b-9785295a817d client request ID: 3eeba58c-cd03-4b39-8109-cd7d06f1728d occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test Engineering want metrics from HCP to be present in Azure Monitor Workspace · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics" Unexpected error: <fmt.wrapError \| 0x13139829f200>: failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-d8dbvxq2vjwr" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c578be5-0e01-4c17-bd7d-12e7070a4b43 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c578be5-0e01-4c17-bd7d-12e7070a4b43", "name": "3c578be5-0e01-4c17-bd7d-12e7070a4b43", "status": "Failed", "startTime": "2026-10-25T11:02:02.600930796Z", "endTime": "2026-10-25T11:31:06.641777786Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m27s ago" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c578be5-0e01-4c17-bd7d-12e7070a4b43 correlation ID: a1430e30-975c-43e0-8415-29dc5c090d8c client request ID: 01e0814b-9e46-446b-922c-7c28ded6e441 occurred fail [github.com/Azure/ARO-HCP/test/e2e/metrics_amw_hcp_ksm.go:87]: failed to create HCP cluster "ksm-hcp-metrics" Unexpected error: <fmt.wrapError \| 0x13139829f200>: failed to create HCP cluster ksm-hcp-metrics: failed waiting for cluster="ksm-hcp-metrics" in resourcegroup="hcp-metrics-d8dbvxq2vjwr" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c578be5-0e01-4c17-bd7d-12e7070a4b43 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c578be5-0e01-4c17-bd7d-12e7070a4b43", "name": "3c578be5-0e01-4c17-bd7d-12e7070a4b43", "status": "Failed", "startTime": "2026-10-25T11:02:02.600930796Z", "endTime": "2026-10-25T11:31:06.641777786Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 22m27s ago" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3c578be5-0e01-4c17-bd7d-12e7070a4b43 correlation ID: a1430e30-975c-43e0-8415-29dc5c090d8c client request ID: 01e0814b-9e46-446b-922c-7c28ded6e441 occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0x299d616e8140>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-bcvsvbp5d8t7" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4d6594c-1ee5-4e01-824a-b335b8b1a671 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4d6594c-1ee5-4e01-824a-b335b8b1a671", "name": "f4d6594c-1ee5-4e01-824a-b335b8b1a671", "status": "Failed", "startTime": "2026-10-25T11:01:32.174992685Z", "endTime": "2026-10-25T11:30:36.037388523Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 24m56s ago; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4d6594c-1ee5-4e01-824a-b335b8b1a671 correlation ID: 88e2d1ef-c0a9-47ed-aaa6-0693add21a52 client request ID: 8ab0d68c-218c-40c0-b734-d3980ba8ecab occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0x299d616e8140>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-bcvsvbp5d8t7" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4d6594c-1ee5-4e01-824a-b335b8b1a671 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4d6594c-1ee5-4e01-824a-b335b8b1a671", "name": "f4d6594c-1ee5-4e01-824a-b335b8b1a671", "status": "Failed", "startTime": "2026-10-25T11:01:32.174992685Z", "endTime": "2026-10-25T11:30:36.037388523Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.39 is Partial (want Completed), started 24m56s ago; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f4d6594c-1ee5-4e01-824a-b335b8b1a671 correlation ID: 88e2d1ef-c0a9-47ed-aaa6-0693add21a52 client request ID: 8ab0d68c-218c-40c0-b734-d3980ba8ecab occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges for cluster version 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x2b59459fc100>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-t7spx49mgg2h" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fde9935e-6ef0-4518-bbc5-60b5d4f16d38 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fde9935e-6ef0-4518-bbc5-60b5d4f16d38", "name": "fde9935e-6ef0-4518-bbc5-60b5d4f16d38", "status": "Failed", "startTime": "2026-10-25T11:01:26.130044072Z", "endTime": "2026-10-25T11:30:36.059795656Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.43 is Partial (want Completed), started 22m37s ago" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fde9935e-6ef0-4518-bbc5-60b5d4f16d38 correlation ID: 1127ed99-2d51-4906-b522-5ba778a6c853 client request ID: eb7f7962-a820-4a0a-acaa-426a2d0c04e3 occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:150]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0x2b59459fc100>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-t7spx49mgg2h" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fde9935e-6ef0-4518-bbc5-60b5d4f16d38 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fde9935e-6ef0-4518-bbc5-60b5d4f16d38", "name": "fde9935e-6ef0-4518-bbc5-60b5d4f16d38", "status": "Failed", "startTime": "2026-10-25T11:01:26.130044072Z", "endTime": "2026-10-25T11:30:36.059795656Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.43 is Partial (want Completed), started 22m37s ago" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fde9935e-6ef0-4518-bbc5-60b5d4f16d38 correlation ID: 1127ed99-2d51-4906-b522-5ba778a6c853 client request ID: eb7f7962-a820-4a0a-acaa-426a2d0c04e3 occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: multiple components; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:93]: failed to create HCP cluster for kusto logs test Unexpected error: <fmt.wrapError \| 0x22535b109360>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-bh6sljsqnwrw" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec3175b8-8979-4854-9d37-2e76e8d09536 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec3175b8-8979-4854-9d37-2e76e8d09536", "name": "ec3175b8-8979-4854-9d37-2e76e8d09536", "status": "Failed", "startTime": "2026-10-25T11:02:03.316597788Z", "endTime": "2026-10-25T11:31:06.803246105Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, cluster-network-operator, redhat-operators-catalog, redhat-marketplace-catalog, ingress-operator, cluster-image-registry-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, cluster-version-operator, cluster-node-tuning-operator, control-plane-pki-operator, azure-cloud-controller-manager, openshift-controller-manager, openshift-route-controller-manager, hosted-cluster-config-operator, olm-operator, packageserver, certified-operators-catalog, catalog-operator, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec3175b8-8979-4854-9d37-2e76e8d09536 correlation ID: 7d2b87e8-723a-49a9-a4cb-8027a1fb4066 client request ID: b5b1bd46-74cb-4267-85b1-e1af986392eb occurred fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:93]: failed to create HCP cluster for kusto logs test Unexpected error: <fmt.wrapError \| 0x22535b109360>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-bh6sljsqnwrw" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec3175b8-8979-4854-9d37-2e76e8d09536 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec3175b8-8979-4854-9d37-2e76e8d09536", "name": "ec3175b8-8979-4854-9d37-2e76e8d09536", "status": "Failed", "startTime": "2026-10-25T11:02:03.316597788Z", "endTime": "2026-10-25T11:31:06.803246105Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-storage-operator, cluster-network-operator, redhat-operators-catalog, redhat-marketplace-catalog, ingress-operator, cluster-image-registry-operator, community-operators-catalog, dns-operator, csi-snapshot-controller-operator, cluster-version-operator, cluster-node-tuning-operator, control-plane-pki-operator, azure-cloud-controller-manager, openshift-controller-manager, openshift-route-controller-manager, hosted-cluster-config-operator, olm-operator, packageserver, certified-operators-catalog, catalog-operator, cluster-policy-controller; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, certified-operators-catalog deployment has 1 unavailable replicas, cluster-image-registry-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, cluster-policy-controller deployment has 2 unavailable replicas, cluster-storage-operator deployment has 1 unavailable replicas, cluster-version-operator deployment has 1 unavailable replicas, community-operators-catalog deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-controller-manager deployment has 2 unavailable replicas, openshift-route-controller-manager deployment has 2 unavailable replicas, packageserver deployment has 3 unavailable replicas, redhat-marketplace-catalog deployment has 1 unavailable replicas, redhat-operators-catalog deployment has 1 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ec3175b8-8979-4854-9d37-2e76e8d09536 correlation ID: 7d2b87e8-723a-49a9-a4cb-8027a1fb4066 client request ID: b5b1bd46-74cb-4267-85b1-e1af986392eb occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-q5hjsn" with version 4.20.42 on candidate channel Unexpected error: <fmt.wrapError \| 0x18f75457b1e0>: failed to create HCP cluster cluster-zstream-4-20-q5hjsn: failed waiting for cluster="cluster-zstream-4-20-q5hjsn" in resourcegroup="rg-zstream-upgrade-4-20-jb8s5lbcxszk" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5400fbc4-9a5a-4a07-8b68-53299b9929f5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5400fbc4-9a5a-4a07-8b68-53299b9929f5", "name": "5400fbc4-9a5a-4a07-8b68-53299b9929f5", "status": "Failed", "startTime": "2026-10-25T11:01:30.073248883Z", "endTime": "2026-10-25T11:30:35.871032188Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.42 is Partial (want Completed), started 23m47s ago" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5400fbc4-9a5a-4a07-8b68-53299b9929f5 correlation ID: cf35d619-4e23-43fc-8aeb-ee92df9074e6 client request ID: 68e2030f-31dd-4592-871e-2e8df326a35b occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:125]: failed to create HCP cluster "cluster-zstream-4-20-q5hjsn" with version 4.20.42 on candidate channel Unexpected error: <fmt.wrapError \| 0x18f75457b1e0>: failed to create HCP cluster cluster-zstream-4-20-q5hjsn: failed waiting for cluster="cluster-zstream-4-20-q5hjsn" in resourcegroup="rg-zstream-upgrade-4-20-jb8s5lbcxszk" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5400fbc4-9a5a-4a07-8b68-53299b9929f5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5400fbc4-9a5a-4a07-8b68-53299b9929f5", "name": "5400fbc4-9a5a-4a07-8b68-53299b9929f5", "status": "Failed", "startTime": "2026-10-25T11:01:30.073248883Z", "endTime": "2026-10-25T11:30:35.871032188Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.42 is Partial (want Completed), started 23m47s ago" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5400fbc4-9a5a-4a07-8b68-53299b9929f5 correlation ID: cf35d619-4e23-43fc-8aeb-ee92df9074e6 client request ID: 68e2030f-31dd-4592-871e-2e8df326a35b occurred ERROR CODE: DeadlineExceeded; detail message cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable; hosted cluster degraded: UnavailableReplicas: [catalog-operator, hosted-cluster-config-operator, olm-operator, openshift-apiserver, packageserver]; provider Microsoft.RedHatOpenShift 09:54:40 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0x1afe2fabc320>: failed to create HCP cluster ea-list7tqb74: failed waiting for cluster="ea-list7tqb74" in resourcegroup="ea-list-8mbx8mstb22c" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/506cd809-e6bc-45d8-aa3a-45042c374b41 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/506cd809-e6bc-45d8-aa3a-45042c374b41", "name": "506cd809-e6bc-45d8-aa3a-45042c374b41", "status": "Failed", "startTime": "2026-10-25T11:02:00.137646561Z", "endTime": "2026-10-25T11:31:06.609068276Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, community-operators-catalog, packageserver, cluster-node-tuning-operator, cluster-storage-operator, ingress-operator, dns-operator, catalog-operator, cluster-network-operator, cluster-version-operator, csi-snapshot-controller-operator, hosted-cluster-config-operator, certified-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, redhat-operators-catalog, cluster-policy-controller, olm-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/506cd809-e6bc-45d8-aa3a-45042c374b41 correlation ID: a86e7a8c-dce2-49be-8312-2dfeeebedc58 client request ID: 03f569a8-daf7-4041-b81b-1efe61e6782e occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0x1afe2fabc320>: failed to create HCP cluster ea-list7tqb74: failed waiting for cluster="ea-list7tqb74" in resourcegroup="ea-list-8mbx8mstb22c" to finish creating: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/506cd809-e6bc-45d8-aa3a-45042c374b41 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/506cd809-e6bc-45d8-aa3a-45042c374b41", "name": "506cd809-e6bc-45d8-aa3a-45042c374b41", "status": "Failed", "startTime": "2026-10-25T11:02:00.137646561Z", "endTime": "2026-10-25T11:31:06.609068276Z", "error": { "code": "DeadlineExceeded", "message": "cluster creation did not complete before the deadline; [clusterServiceClusterStatus] cluster service is installing; [hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-image-registry-operator, community-operators-catalog, packageserver, cluster-node-tuning-operator, cluster-storage-operator, ingress-operator, dns-operator, catalog-operator, cluster-network-operator, cluster-version-operator, csi-snapshot-controller-operator, hosted-cluster-config-operator, certified-operators-catalog, openshift-controller-manager, openshift-route-controller-manager, redhat-operators-catalog, cluster-policy-controller, olm-operator, redhat-marketplace-catalog; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, hosted-cluster-config-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, openshift-apiserver deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j7524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/506cd809-e6bc-45d8-aa3a-45042c374b41 correlation ID: a86e7a8c-dce2-49be-8312-2dfeeebedc58 client request ID: 03f569a8-daf7-4041-b81b-1efe61e6782e occurred alert (4) alert [svc] FluentBitIngestionPaused fired 09:54:40 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FluentBitIngestionPaused does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-10-25T11:48:54Z Ended: 2026-10-25T11:58:53Z Severity: Sev3 Labels: alertname="FluentBitIngestionPaused", cluster="ci01-j7524864-mgmt-2", component="monitoring-infrastructure", pod="arobit-forwarder-q4jd5", region="westus3", severity="warning" Description: Fluent Bit pod arobit-forwarder-q4jd5 on cluster ci01-j7524864-mgmt-2 has paused collecting new log data for at least 5 minutes. Ingestion pauses when Fluent Bit\'s internal memory or storage buffers are full, typically caused by backpressure from a slow or failing output. Investigate the Fluent Bit logs for the specific error details and check the Kusto instance health. alert [hcp] KubePodNotReady fired 09:54:40 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 34 time(s) (3 unknown, 31 known) Firing 1: State: Resolved Started: 2026-10-25T11:42:42Z Ended: 2026-10-25T11:50:40Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tlp73656llsbjr851ct21vm7ag358hj-cilium-cl", pod="azure-file-csi-driver-controller-66f6c89876-l6fts", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tlp73656llsbjr851ct21vm7ag358hj-cilium-cl/azure-file-csi-driver-controller-66f6c89876-l6fts has been in a non-ready state for longer than 5 minutes. Firing 2: State: Resolved Started: 2026-10-25T11:44:40Z Ended: 2026-10-25T11:50:40Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tlp73656llsbjr851ct21vm7ag358hj-cilium-cl", pod="multus-admission-controller-58547c8cf6-lmftl", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tlp73656llsbjr851ct21vm7ag358hj-cilium-cl/multus-admission-controller-58547c8cf6-lmftl has been in a non-ready state for longer than 5 minutes. Firing 3: State: Resolved Started: 2026-10-25T11:53:40Z Ended: 2026-10-25T11:59:42Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2tlp70no771drpm6jf2csmj08ibs461h-s2r6f9c0k1q1q9w", pod="etcd-0", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2tlp70no771drpm6jf2csmj08ibs461h-s2r6f9c0k1q1q9w/etcd-0 has been in a non-ready state for longer than 5 minutes. alert [svc] KubePodCrashLooping fired 09:54:40 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubePodCrashLooping does not fire · suite aro-hcp-tests Show raw failure alert fired 8 time(s) Firing 1: State: Resolved Started: 2026-10-25T11:31:30Z Ended: 2026-10-25T11:58:30Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="manager", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="secret-sync-controller", pod="secrets-store-sync-controller-manager-7f4b9bc5df-czwqq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="76e86419-4829-44c0-a136-817aadd188d4" Description: Pod secret-sync-controller/secrets-store-sync-controller-manager-7f4b9bc5df-czwqq (manager) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-25T11:31:30Z Ended: 2026-10-25T11:58:31Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="manager", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="secret-sync-controller", pod="secrets-store-sync-controller-manager-7f4b9bc5df-czwqq", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="76e86419-4829-44c0-a136-817aadd188d4" Description: Pod secret-sync-controller/secrets-store-sync-controller-manager-7f4b9bc5df-czwqq (manager) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Resolved Started: 2026-10-25T11:37:32Z Ended: 2026-10-25T11:48:32Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="governance-policy-propagator", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="grc-policy-propagator-5fc7fb4677-67tll", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e5593f36-ff37-4a56-9501-050ee9a5f489" Description: Pod multicluster-engine/grc-policy-propagator-5fc7fb4677-67tll (governance-policy-propagator) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Resolved Started: 2026-10-25T11:37:32Z Ended: 2026-10-25T11:48:32Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="governance-policy-propagator", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="grc-policy-propagator-5fc7fb4677-67tll", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e5593f36-ff37-4a56-9501-050ee9a5f489" Description: Pod multicluster-engine/grc-policy-propagator-5fc7fb4677-67tll (governance-policy-propagator) is in waiting state (reason: "CrashLoopBackOff"). Firing 5: State: Resolved Started: 2026-10-25T11:39:32Z Ended: 2026-10-25T11:48:32Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="governance-policy-propagator", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="grc-policy-propagator-5fc7fb4677-j5hzx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="eabaed55-b955-4173-b78d-fec0366c15f2" Description: Pod multicluster-engine/grc-policy-propagator-5fc7fb4677-j5hzx (governance-policy-propagator) is in waiting state (reason: "CrashLoopBackOff"). Firing 6: State: Resolved Started: 2026-10-25T11:39:32Z Ended: 2026-10-25T11:48:32Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="governance-policy-propagator", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="grc-policy-propagator-5fc7fb4677-j5hzx", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="eabaed55-b955-4173-b78d-fec0366c15f2" Description: Pod multicluster-engine/grc-policy-propagator-5fc7fb4677-j5hzx (governance-policy-propagator) is in waiting state (reason: "CrashLoopBackOff"). Firing 7: State: Resolved Started: 2026-10-25T11:41:31Z Ended: 2026-10-25T11:52:33Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="ocm-controller", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="ocm-controller-859484fc49-pj8f8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a3ea24fa-145a-42ea-812b-1fe46f10bfd1" Description: Pod multicluster-engine/ocm-controller-859484fc49-pj8f8 (ocm-controller) is in waiting state (reason: "CrashLoopBackOff"). Firing 8: State: Resolved Started: 2026-10-25T11:41:31Z Ended: 2026-10-25T11:52:33Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="ocm-controller", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/services-j7524864", microsoft.metrics_include_label="service", namespace="multicluster-engine", pod="ocm-controller-859484fc49-pj8f8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="a3ea24fa-145a-42ea-812b-1fe46f10bfd1" Description: Pod multicluster-engine/ocm-controller-859484fc49-pj8f8 (ocm-controller) is in waiting state (reason: "CrashLoopBackOff"). alert [hcp] KubePodCrashLooping fired 09:54:40 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire · suite aro-hcp-tests Show raw failure alert fired 3 time(s) Firing 1: State: Resolved Started: 2026-10-25T11:41:40Z Ended: 2026-10-25T11:51:40Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="manager", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/hcps-j7524864", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlp722bk3stbjb74e8i42b30lftatrs-ingress-cert-le", pod="capi-provider-6d7bfdf44-24wm6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5602ef00-22ff-45dd-b3cb-39a5ba53e966" Description: Pod ocm-arohcpci01-2tlp722bk3stbjb74e8i42b30lftatrs-ingress-cert-le/capi-provider-6d7bfdf44-24wm6 (manager) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-10-25T11:41:40Z Ended: 2026-10-25T11:51:41Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="manager", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/hcps-j7524864", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlp722bk3stbjb74e8i42b30lftatrs-ingress-cert-le", pod="capi-provider-6d7bfdf44-24wm6", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="5602ef00-22ff-45dd-b3cb-39a5ba53e966" Description: Pod ocm-arohcpci01-2tlp722bk3stbjb74e8i42b30lftatrs-ingress-cert-le/capi-provider-6d7bfdf44-24wm6 (manager) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Resolved Started: 2026-10-25T11:43:40Z Ended: 2026-10-25T11:51:41Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j7524864-mgmt-2", component="kubernetes-infrastructure", container="manager", endpoint="http", environment="ci01", instance="10.128.64.204:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7524864/providers/microsoft.monitor/accounts/hcps-j7524864", microsoft.metrics_include_label="service", namespace="ocm-arohcpci01-2tlp733dhbjgega0lnvcjvgg1grv9tf4-i6w1d6x7y2t6h2v", pod="capi-provider-8f7b799d8-49zjj", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="706780f8-3192-4947-b780-05f86e3d1b53" Description: Pod ocm-arohcpci01-2tlp733dhbjgega0lnvcjvgg1grv9tf4-i6w1d6x7y2t6h2v/capi-provider-8f7b799d8-49zjj (manager) is in waiting state (reason: "CrashLoopBackOff").
09:53:48	2h 28m 13s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 8a24997	canadacentral	alert, e2e	failed	⚠#7712 (closed)	3	e2e (2) DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded 09:53:48 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20 Unexpected error: <fmt.wrapError \| 0xc4f5ed30b40>: DNS for route host agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded { msg: "DNS for route host agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded", err: <fmt.wrapError \| 0xc4f5f2da4a0>{ msg: "DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded", err: <context.deadlineExceededError>{}, }, } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:201]: failed to verify simple web app on back-level cluster version 4.20 Unexpected error: <fmt.wrapError \| 0xc4f5ed30b40>: DNS for route host agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded { msg: "DNS for route host agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded", err: <fmt.wrapError \| 0xc4f5f2da4a0>{ msg: "DNS for agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-jfjrs.apps.aro.a7b4y7y9a8k0c1j.bzjc.j0985984.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded", err: <context.deadlineExceededError>{}, }, } occurred missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10782, acm-agent=0 09:53:48 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.000s. Expected success, but got an error: <errors.errorString \| 0x2f3433d553d0>: missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10782, acm-agent=0 { s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10782, acm-agent=0", } fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:118]: Timed out after 600.000s. Expected success, but got an error: <errors.errorString \| 0x2f3433d553d0>: missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10782, acm-agent=0 { s: "missing expected log sources: []; detailedInfraOrchestrationLogs expected at least one row per required source, missing: [acm-agent]; observed counts: hypershift=10782, acm-agent=0", } alert (1) alert [svc] KubePodCrashLooping fired 09:53:48 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubePodCrashLooping does not fire · suite aro-hcp-tests Show raw failure alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-10-25T11:00:02Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0985984-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.246:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985984/providers/microsoft.monitor/accounts/services-j0985984", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-w7slc", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="cba8a3e2-ad50-446e-87a6-2c6f376d3905" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-w7slc (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Fired (not resolved) Started: 2026-10-25T11:02:02Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0985984-mgmt-2", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.64.246:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985984/providers/microsoft.monitor/accounts/services-j0985984", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-w7slc", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="cba8a3e2-ad50-446e-87a6-2c6f376d3905" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-w7slc (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 3: State: Fired (not resolved) Started: 2026-10-25T11:02:02Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0985984-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.9:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985984/providers/microsoft.monitor/accounts/services-j0985984", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-qdxb4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f78cf57b-d3eb-455a-89b9-36c6a269ad3f" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-qdxb4 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff"). Firing 4: State: Fired (not resolved) Started: 2026-10-25T11:04:02Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0985984-mgmt-1", component="kubernetes-infrastructure", container="hypershift-addon-agent", endpoint="http", environment="ci01", instance="10.128.65.9:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0985984/providers/microsoft.monitor/accounts/services-j0985984", microsoft.metrics_include_label="service", namespace="open-cluster-management-agent-addon", pod="hypershift-addon-agent-76ffb6fdb8-qdxb4", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="f78cf57b-d3eb-455a-89b9-36c6a269ad3f" Description: Pod open-cluster-management-agent-addon/hypershift-addon-agent-76ffb6fdb8-qdxb4 (hypershift-addon-agent) is in waiting state (reason: "CrashLoopBackOff").
09:44:02	2h 35m 30s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 1de2577	westus3	e2e	failed	#7657 (open)	1	e2e (1) ERROR CODE: DeadlineExceeded; detail message node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is <count>, want <count>; node pool condition AllNodesHealthy is False: <count> of <count> machines are not healthy Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; node pool condition AllMachinesReady is False: <count> of <count> machines are not ready Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist Machine <node>: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Compute/virtualMachines/<node> to exist; provider Microsoft.RedHatOpenShift 09:44:02 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster Unexpected error: <fmt.wrapError \| 0x2a5fafb1fdc0>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-dhxpqc2757nk" to finish creating: GET https://rp.j9438720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415", "name": "995ad93d-44a1-4aae-9f3c-cef4914b4415", "status": "Failed", "startTime": "2026-10-25T11:09:27.17231924Z", "endTime": "2026-10-25T11:28:33.403436193Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8 to exist\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8 to exist\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt to exist\n" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j9438720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415 correlation ID: d01faed1-7c94-4749-8892-78ade8753a54 client request ID: 5d2be0a3-7126-4924-b3ac-5c1c229ea51d occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster Unexpected error: <fmt.wrapError \| 0x2a5fafb1fdc0>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-dhxpqc2757nk" to finish creating: GET https://rp.j9438720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeadlineExceeded -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415", "name": "995ad93d-44a1-4aae-9f3c-cef4914b4415", "status": "Failed", "startTime": "2026-10-25T11:09:27.17231924Z", "endTime": "2026-10-25T11:28:33.403436193Z", "error": { "code": "DeadlineExceeded", "message": "node pool creation did not complete before the deadline; [clusterServiceNodePoolStatus] cluster service node pool is installing; [hypershiftNodePool] hypershift NodePool status replicas is 0, want 2; node pool condition AllNodesHealthy is False: 2 of 2 machines are not healthy\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8 to exist\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt: InspectionFailed: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt to exist\n; node pool condition AllMachinesReady is False: 2 of 2 machines are not ready\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8: ReadyUnknown: NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-kjbw8 to exist\nMachine w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt: ReadyUnknown: * NodeHealthy: Waiting for a Node with spec.providerID azure:///subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dhxpqc2757nk--managed/providers/Microsoft.Compute/virtualMachines/w1u6b4w7n6r4n6j-np-1-tvg6z-njbxt to exist\n" } } -------------------------------------------------------------------------------- <Azure error internals omitted> Azure request IDs: GET https://rp.j9438720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/995ad93d-44a1-4aae-9f3c-cef4914b4415 correlation ID: d01faed1-7c94-4749-8892-78ade8753a54 client request ID: 5d2be0a3-7126-4924-b3ac-5c1c229ea51d occurred
09:28:34	1h 9m 14s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 1f5a50b	westus3	provision	failed	★#7662 (open)	1	provision (1) failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1 context deadline exceeded 09:28:34 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SecretSyncController/management/deploy (stamp=2) · suite templatize-pipeline Show raw failure time=2026-10-25T10:27:22.014Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 description="Step deploy\n Kind: Helm\n" time=2026-10-25T10:27:22.644Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:b98f9f1c3c603f0e3ee70cd3d70746d8265b8d425df5b71f23c4b0dcb3eb979c imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:b451542dd17a9b1b1b400fa1ca768a492720b87b696f373f0494217d855aee8c imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]" time=2026-10-25T10:27:22.644Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-25T10:27:22.918Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 namespace=secret-sync-controller time=2026-10-25T10:27:23.279Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 dryRun=false time=2026-10-25T10:27:23.279Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-25T10:27:23.445Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 time=2026-10-25T10:32:28.984Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T10:32:28.985Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=2 err="stamp 2: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
07:28:51	1h 1m 39s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 1f5a50b	westus3	provision	failed	★#7662 (open)	1	provision (1) failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1 context deadline exceeded 07:28:51 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SecretSyncController/management/deploy (stamp=1) · suite templatize-pipeline Show raw failure time=2026-10-25T08:14:44.299Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 description="Step deploy\n Kind: Helm\n" time=2026-10-25T08:14:45.051Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:b98f9f1c3c603f0e3ee70cd3d70746d8265b8d425df5b71f23c4b0dcb3eb979c imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:b451542dd17a9b1b1b400fa1ca768a492720b87b696f373f0494217d855aee8c imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]" time=2026-10-25T08:14:45.051Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T08:14:45.565Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 namespace=secret-sync-controller time=2026-10-25T08:14:45.984Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 dryRun=false time=2026-10-25T08:14:45.984Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T08:14:46.156Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T08:19:54.254Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T08:19:54.254Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
06:29:29	44m 51s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 1f5a50b	canadacentral	provision	failed	★#7662 (open)	1	provision (1) failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1 context deadline exceeded 06:29:29 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SecretSyncController/management/deploy (stamp=1) · suite templatize-pipeline Show raw failure time=2026-10-25T06:58:33.435Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 description="Step deploy\n Kind: Helm\n" time=2026-10-25T06:58:33.964Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:b98f9f1c3c603f0e3ee70cd3d70746d8265b8d425df5b71f23c4b0dcb3eb979c imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:b451542dd17a9b1b1b400fa1ca768a492720b87b696f373f0494217d855aee8c imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]" time=2026-10-25T06:58:33.964Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T06:58:34.148Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 namespace=secret-sync-controller time=2026-10-25T06:58:34.349Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 dryRun=false time=2026-10-25T06:58:34.350Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T06:58:34.404Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T07:03:38.507Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T07:03:38.507Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
05:13:31	59m 11s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 1f5a50b	centralus	provision	failed	★#7662 (open)	1	provision (1) failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1 context deadline exceeded 05:13:31 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.SecretSyncController/management/deploy (stamp=1) · suite templatize-pipeline Show raw failure time=2026-10-25T06:01:12.761Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 description="Step deploy\n Kind: Helm\n" time=2026-10-25T06:01:13.196Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 values="map[controllerName:secrets-store-sync-controller-manager image:arohcpsvcdev.azurecr.io/secrets-store-sync/controller@sha256:b98f9f1c3c603f0e3ee70cd3d70746d8265b8d425df5b71f23c4b0dcb3eb979c imagePullPolicy:IfNotPresent metricsPort:8085 providerContainer:[map[args:[--endpoint=unix:///provider/azure.sock --construct-pem-chain=true --healthz-port=8989 --healthz-path=/healthz --healthz-timeout=5s] image:mcr.microsoft.com/oss/v2/azure/secrets-store/provider-azure@sha256:b451542dd17a9b1b1b400fa1ca768a492720b87b696f373f0494217d855aee8c imagePullPolicy:IfNotPresent livenessProbe:map[failureThreshold:3 httpGet:map[path:/healthz port:8989] initialDelaySeconds:5 periodSeconds:30 timeoutSeconds:10] name:provider-azure-installer ports:[map[containerPort:8898 name:metrics protocol:TCP]] resources:map[limits:map[memory:unlimited] requests:map[cpu:50m memory:100Mi]] securityContext:map[allowPrivilegeEscalation:false capabilities:map[drop:[ALL]] readOnlyRootFilesystem:true runAsUser:0] volumeMounts:[map[mountPath:/provider name:providervol]]]] tokenRequestAudience:[map[audience:<nil>]] validatingAdmissionPolicies:map[allowedSecretTypes:[Opaque kubernetes.io/basic-auth bootstrap.kubernetes.io/token kubernetes.io/dockerconfigjson kubernetes.io/dockercfg kubernetes.io/ssh-auth kubernetes.io/tls] applyPolicies:true deniedSecretTypes:[kubernetes.io/service-account-token]]]" time=2026-10-25T06:01:13.196Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T06:01:13.397Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 namespace=secret-sync-controller time=2026-10-25T06:01:13.537Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 dryRun=false time=2026-10-25T06:01:13.537Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T06:01:13.638Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 time=2026-10-25T06:06:16.971Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded" time=2026-10-25T06:06:16.971Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.SecretSyncController resourceGroup=management step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/secret-sync-controller/secrets-store-sync-controller-manager not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
01:34:08	2h 8m 8s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head ed0da09	centralus	n/a	passed	#6452 (open)	0	n/a