CIHealth Run Log

Environment: DEV

Time (UTC)	Runtime	Job	Region	Failed at	Result	PR	Failed tests	Details
23:29:40	2h 0m 2s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3546f07	westus3	n/a	passed	#7053 (open)	0	n/a
23:02:54	2h 10m 2s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 115a341	canadacentral	alert	failed	#6821 (open)	1	alert (1) alert [svc] ServiceMemoryDrift fired 23:02:54 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-23T00:53:07Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j7759104-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-pn2sd", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-pn2sd (namespace arobit) on cluster ci01-j7759104-svc has averaged 127.3% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
23:02:41	2h 9m 32s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 87e40a0	canadacentral	alert	failed	#6922 (open)	1	alert (1) alert [svc] FrontendPathLatency fired 23:02:41 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-23T00:42:32Z Ended: 2026-09-23T00:53:32Z Severity: Sev3 Labels: alertname="FrontendPathLatency", cluster="ci01-j6922368-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters", severity="warning" Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters exceeded 1 second over the past 30 minutes (cluster ci01-j6922368-svc).
23:02:40	1h 21m 12s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c99d95a	centralus	alert, provision	failed	#6820 (open)	4	provision (1) failed to roll out Helm release: context deadline exceeded 23:02:40 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/arobit · suite templatize-pipeline Show raw failure time=2026-09-22T23:47:19.977Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n" time=2026-09-22T23:47:20.376Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j3566464-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:centralus role:hcp-underlay-ci01-j3566464-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:3739bfae-90c4-46a6-9e9a-788bc6da855b region:centralus secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#3739bfae-90c4-46a6-9e9a-788bc6da855b INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]" time=2026-09-22T23:47:20.376Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T23:47:20.465Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit time=2026-09-22T23:47:20.652Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false time=2026-09-22T23:47:20.652Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T23:47:20.723Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T23:57:23.239Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded" time=2026-09-22T23:57:23.240Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded" time=2026-09-22T23:58:23.243Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n" time=2026-09-22T23:58:23.878Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j3566464-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:centralus role:hcp-underlay-ci01-j3566464-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:3739bfae-90c4-46a6-9e9a-788bc6da855b region:centralus secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#3739bfae-90c4-46a6-9e9a-788bc6da855b INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]" time=2026-09-22T23:58:23.878Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T23:58:24.112Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit time=2026-09-22T23:58:24.439Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T23:58:24.439Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T23:58:24.639Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1 time=2026-09-22T23:58:25.408Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T23:58:25.408Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder time=2026-09-22T23:58:25.669Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder time=2026-09-22T23:58:26.134Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder time=2026-09-22T23:58:26.496Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder time=2026-09-22T23:58:26.679Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder time=2026-09-22T23:58:26.994Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder time=2026-09-22T23:58:27.236Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder time=2026-09-22T23:58:27.460Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false time=2026-09-22T23:58:27.460Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T23:58:27.560Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1 time=2026-09-23T00:08:29.375Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded" time=2026-09-23T00:08:29.375Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded" time=2026-09-23T00:09:29.384Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n" time=2026-09-23T00:09:29.909Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j3566464-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:centralus role:hcp-underlay-ci01-j3566464-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:3739bfae-90c4-46a6-9e9a-788bc6da855b region:centralus secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#3739bfae-90c4-46a6-9e9a-788bc6da855b INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]" time=2026-09-23T00:09:29.909Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-23T00:09:30.097Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit time=2026-09-23T00:09:30.328Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-23T00:09:30.328Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-23T00:09:30.580Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2 time=2026-09-23T00:09:31.341Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-23T00:09:31.341Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder time=2026-09-23T00:09:31.535Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder time=2026-09-23T00:09:31.879Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder time=2026-09-23T00:09:32.169Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder time=2026-09-23T00:09:32.570Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder time=2026-09-23T00:09:32.841Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder time=2026-09-23T00:09:33.156Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder time=2026-09-23T00:09:33.376Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false time=2026-09-23T00:09:33.376Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-23T00:09:33.452Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2 time=2026-09-23T00:17:19.905Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded" time=2026-09-23T00:17:19.906Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded" alert (3) alert [svc] ArobitForwarderJobUp fired 23:02:40 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ArobitForwarderJobUp does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-23T00:04:14Z Severity: Sev3 Labels: alertname="ArobitForwarderJobUp", cluster="ci01-j3566464-svc", component="monitoring-infrastructure", region="centralus", severity="warning" Description: The Arobit forwarder metrics endpoint on cluster ci01-j3566464-svc has been unreachable for 15 minutes. alert [svc] KubeDaemonSetRolloutStuck fired 23:02:40 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubeDaemonSetRolloutStuck does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-23T00:03:29Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j3566464-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.65:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3566464/providers/microsoft.monitor/accounts/services-j3566464", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-lcpcf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes. Firing 2: State: Fired (not resolved) Started: 2026-09-23T00:03:29Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j3566464-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.65:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3566464/providers/microsoft.monitor/accounts/services-j3566464", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-lcpcf", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes. alert [svc] KubePodNotReady fired 23:02:40 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T23:53:28Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3566464-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-5x4b6", region="centralus", severity="warning" Description: Pod arobit/arobit-forwarder-5x4b6 has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-09-22T23:53:28Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3566464-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-56wq6", region="centralus", severity="warning" Description: Pod arobit/arobit-forwarder-56wq6 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-09-22T23:53:28Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3566464-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-njbgt", region="centralus", severity="warning" Description: Pod arobit/arobit-forwarder-njbgt has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-09-22T23:53:28Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3566464-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-dkjc6", region="centralus", severity="warning" Description: Pod arobit/arobit-forwarder-dkjc6 has been in a non-ready state for longer than 5 minutes.
23:02:36	1h 52m 41s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head f79a2b4 · merge f79a2b4	westus3	alert, e2e	failed	#7064 (merged)	6	e2e (1) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 23:02:36 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-bg4xww with version 4.21.34 Unexpected error: <fmt.wrapError \| 0xc0016eec80>: failed to create HCP cluster np-version-upgrade-cluster-bg4xww: failed waiting for cluster="np-version-upgrade-cluster-bg4xww" in resourcegroup="rg-np-version-upgrade-bg4xww-fxbpl7ffx59q" to finish creating: GET https://rp.j3962368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0f4e4436-53e4-4539-ab16-39eefac91dac -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0f4e4436-53e4-4539-ab16-39eefac91dac", "name": "0f4e4436-53e4-4539-ab16-39eefac91dac", "status": "Failed", "startTime": "2026-09-22T23:45:41.203501418Z", "endTime": "2026-09-23T00:14:41.784000482Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-bg4xww with version 4.21.34 Unexpected error: <fmt.wrapError \| 0xc0016eec80>: failed to create HCP cluster np-version-upgrade-cluster-bg4xww: failed waiting for cluster="np-version-upgrade-cluster-bg4xww" in resourcegroup="rg-np-version-upgrade-bg4xww-fxbpl7ffx59q" to finish creating: GET https://rp.j3962368.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0f4e4436-53e4-4539-ab16-39eefac91dac -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0f4e4436-53e4-4539-ab16-39eefac91dac", "name": "0f4e4436-53e4-4539-ab16-39eefac91dac", "status": "Failed", "startTime": "2026-09-22T23:45:41.203501418Z", "endTime": "2026-09-23T00:14:41.784000482Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (5) alert [hcp] KubePodNotReady fired 23:02:36 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) (1 unknown, 1 known) Firing 1: State: Resolved Started: 2026-09-23T00:19:53Z Ended: 2026-09-23T00:25:51Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t0ca0g6h5bft9u4f3cil433bkiopk5i-t0j1r2j4w5n7o8z", pod="router-77497b854-mxvbx", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2t0ca0g6h5bft9u4f3cil433bkiopk5i-t0j1r2j4w5n7o8z/router-77497b854-mxvbx has been in a non-ready state for longer than 5 minutes. alert [hcp] KubePodCrashLooping fired 23:02:36 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-09-23T00:08:48Z Ended: 2026-09-23T00:37:50Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/hcps-j3962368", namespace="ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e1e0ac9c-ae62-4294-a73a-42eb6033d5d4" Description: Pod ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-09-23T00:09:48Z Ended: 2026-09-23T00:37:49Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/hcps-j3962368", namespace="ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e1e0ac9c-ae62-4294-a73a-42eb6033d5d4" Description: Pod ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff"). alert [svc] KubeDeploymentReplicasMismatch fired 23:02:36 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 5 time(s) Firing 1: State: Resolved Started: 2026-09-23T00:19:55Z Ended: 2026-09-23T00:26:51Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/services-j3962368", namespace="klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-09-23T00:19:55Z Ended: 2026-09-23T00:25:51Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/services-j3962368", namespace="klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes. Firing 3: State: Resolved Started: 2026-09-23T00:19:55Z Ended: 2026-09-23T00:26:52Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/services-j3962368", namespace="klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes. Firing 4: State: Resolved Started: 2026-09-23T00:19:55Z Ended: 2026-09-23T00:25:51Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/services-j3962368", namespace="klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes. Firing 5: State: Resolved Started: 2026-09-23T00:19:56Z Ended: 2026-09-23T00:25:51Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/services-j3962368", namespace="klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2t0ca9hj1ibgehdpkuutjoor5mhimmon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes. alert [hcp] KubeDeploymentReplicasMismatch fired 23:02:36 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-09-23T00:20:53Z Ended: 2026-09-23T00:34:52Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/hcps-j3962368", namespace="ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-09-23T00:20:53Z Ended: 2026-09-23T00:39:52Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/hcps-j3962368", namespace="ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert [hcp] KubeStatefulSetReplicasMismatch fired 23:02:36 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-09-23T00:05:55Z Ended: 2026-09-23T00:33:57Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/hcps-j3962368", namespace="ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-09-23T00:06:59Z Ended: 2026-09-23T00:37:57Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j3962368-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.132:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3962368/providers/microsoft.monitor/accounts/hcps-j3962368", namespace="ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n", pod="arohcp-monitor-kube-state-metrics-7955558448-r9h6m", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2t0ca9hj1ibgehdpkuutjoor5mhimmon-s3k7g2g7e0s8u2n/etcd has not matched the expected number of replicas for longer than 15 minutes.
22:49:08	2h 2m 59s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 0a81896 · merge 0a81896	westus3	alert	failed	#7089 (merged)	1	alert (1) alert [svc] FrontendPathLatency fired 22:49:08 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-23T00:20:40Z Ended: 2026-09-23T00:32:36Z Severity: Sev3 Labels: alertname="FrontendPathLatency", cluster="ci01-j7723520-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters", severity="warning" Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters exceeded 1 second over the past 30 minutes (cluster ci01-j7723520-svc).
22:46:36	2h 25m 45s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head ec8ebc7	canadacentral	alert, e2e, unknown	failed	⚠#6359 (open)	5	e2e (3) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 22:46:36 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001f04078>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-xvhgrrnh7hjx" to finish deleting: GET https://rp.j2688256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/177d426f-dd32-4799-8b9b-eb1aaa182ca9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/177d426f-dd32-4799-8b9b-eb1aaa182ca9", "name": "177d426f-dd32-4799-8b9b-eb1aaa182ca9", "status": "Failed", "startTime": "2026-09-22T23:54:44.577772615Z", "endTime": "2026-09-23T00:18:50.338452518Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001f04078>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="basic-hcp-cluster2" in resourcegroup="customer-rg-xvhgrrnh7hjx" to finish deleting: GET https://rp.j2688256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/177d426f-dd32-4799-8b9b-eb1aaa182ca9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/177d426f-dd32-4799-8b9b-eb1aaa182ca9", "name": "177d426f-dd32-4799-8b9b-eb1aaa182ca9", "status": "Failed", "startTime": "2026-09-22T23:54:44.577772615Z", "endTime": "2026-09-23T00:18:50.338452518Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 22:46:36 UTC · failed at e2e · test Customer should be able to cordon, drain, and uncordon a node in an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0012b0090>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-mm2mhl4ksfsn" to finish deleting: GET https://rp.j2688256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d1c688d-941e-4b98-9435-e64aad496c66 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d1c688d-941e-4b98-9435-e64aad496c66", "name": "5d1c688d-941e-4b98-9435-e64aad496c66", "status": "Failed", "startTime": "2026-09-23T00:01:29.977367864Z", "endTime": "2026-09-23T00:25:30.446435885Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0c9ahk4qlhegp42spvj33akbiomek9 still exists (deletion dispatched at 2026-09-23T00:01:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0012b0090>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-mm2mhl4ksfsn" to finish deleting: GET https://rp.j2688256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d1c688d-941e-4b98-9435-e64aad496c66 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d1c688d-941e-4b98-9435-e64aad496c66", "name": "5d1c688d-941e-4b98-9435-e64aad496c66", "status": "Failed", "startTime": "2026-09-23T00:01:29.977367864Z", "endTime": "2026-09-23T00:25:30.446435885Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0c9ahk4qlhegp42spvj33akbiomek9 still exists (deletion dispatched at 2026-09-23T00:01:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 22:46:36 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0013740c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nzhgrm9s9j4m" to finish deleting: GET https://rp.j2688256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/08842c66-3ce5-48d7-9b8b-2b725af2fdb6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/08842c66-3ce5-48d7-9b8b-2b725af2fdb6", "name": "08842c66-3ce5-48d7-9b8b-2b725af2fdb6", "status": "Failed", "startTime": "2026-09-23T00:01:27.518475459Z", "endTime": "2026-09-23T00:25:30.539881421Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0c8kgo0dundgseri9pe4b9l9rg3k7o still exists (deletion dispatched at 2026-09-23T00:01:28Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0013740c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nzhgrm9s9j4m" to finish deleting: GET https://rp.j2688256.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/08842c66-3ce5-48d7-9b8b-2b725af2fdb6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/08842c66-3ce5-48d7-9b8b-2b725af2fdb6", "name": "08842c66-3ce5-48d7-9b8b-2b725af2fdb6", "status": "Failed", "startTime": "2026-09-23T00:01:27.518475459Z", "endTime": "2026-09-23T00:25:30.539881421Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0c8kgo0dundgseri9pe4b9l9rg3k7o still exists (deletion dispatched at 2026-09-23T00:01:28Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [hcp] userJourneySwiftRouterAvailability1h5m fired 22:46:36 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert userJourneySwiftRouterAvailability1h5m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T23:53:49Z Ended: 2026-09-23T00:03:49Z Severity: Sev3 Labels: alertname="userJourneySwiftRouterAvailability1h5m", burn_rate_tier="fast", cluster="ci01-j2688256-mgmt-1", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2688256/providers/microsoft.monitor/accounts/hcps-j2688256", region="canadacentral", severity="3", short_window="5m", slo="swift-router-availability" Description: SWIFT router-capacity availability is consuming its error budget at the fast burn rate. unknown (1) Process did not finish before 15m0s timeout 22:46:36 UTC · failed at unknown · test Job run should complete before timeout · suite prowjob-junit Show raw failure {"component":"entrypoint","file":"sigs.k8s.io/prow/pkg/entrypoint/run.go:178","func":"sigs.k8s.io/prow/pkg/entrypoint.Options.ExecuteProcess","level":"error","msg":"Process did not finish before 15m0s timeout","severity":"error","time":"2026-09-23T00:52:30Z"}
22:03:30	1h 54m 2s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head ef82149 · merge ef82149	centralus	n/a	passed	#7033 (merged)	0	n/a
21:32:48	1h 57m 23s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 571559a · merge 571559a	canadacentral	n/a	passed	#6516 (merged)	0	n/a
21:12:09	1h 59m 37s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3546f07	canadacentral	e2e	failed	#7053 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 21:12:09 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-gjkk89" for cluster "cluster-candidate-4-22-gjkk89" Unexpected error: <fmt.wrapError \| 0xc0011f1c80>: failed to create NodePool np-gjkk89: failed waiting for nodepool="np-gjkk89" for cluster "cluster-candidate-4-22-gjkk89" in resourcegroup="rg-candidate-4-22-pm8qfxg2znfc" to finish creating: GET https://rp.j5377280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76e0adbd-39c6-46a6-83d0-b884d2078d83 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76e0adbd-39c6-46a6-83d0-b884d2078d83", "name": "76e0adbd-39c6-46a6-83d0-b884d2078d83", "status": "Failed", "startTime": "2026-09-22T22:15:39.485216693Z", "endTime": "2026-09-22T22:34:46.302814783Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-gjkk89" for cluster "cluster-candidate-4-22-gjkk89" Unexpected error: <fmt.wrapError \| 0xc0011f1c80>: failed to create NodePool np-gjkk89: failed waiting for nodepool="np-gjkk89" for cluster "cluster-candidate-4-22-gjkk89" in resourcegroup="rg-candidate-4-22-pm8qfxg2znfc" to finish creating: GET https://rp.j5377280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76e0adbd-39c6-46a6-83d0-b884d2078d83 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/76e0adbd-39c6-46a6-83d0-b884d2078d83", "name": "76e0adbd-39c6-46a6-83d0-b884d2078d83", "status": "Failed", "startTime": "2026-09-22T22:15:39.485216693Z", "endTime": "2026-09-22T22:34:46.302814783Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred
21:10:06	2h 14m 7s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 5126a8a	canadacentral	n/a	passed	#7054 (open)	0	n/a
21:01:29	2h 4m 53s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head e8df860 · merge e8df860	westus3	alert	failed	★#5846 (merged)	1	alert (1) alert [svc] userJourneyFrontendLatency6h30m fired 21:01:29 UTC · failed at alert · test [aro-hcp-observability] [svc] alert userJourneyFrontendLatency6h30m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T22:54:46Z Severity: Sev3 Labels: alertname="userJourneyFrontendLatency6h30m", cluster="ci01-j4395136-svc", component="slo", long_window="6h", region="westus3", severity="3", short_window="30m", slo="frontend-latency" Description: More than 6% of Frontend requests on cluster ci01-j4395136-svc exceeded 1s on both the 6h and 30m windows (6x burn of the 1% latency budget).
21:01:10	1h 25m 46s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c99d95a	westus3	alert, provision	failed	#6820 (open)	4	provision (1) failed to roll out Helm release: context deadline exceeded 21:01:10 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/arobit · suite templatize-pipeline Show raw failure time=2026-09-22T21:50:03.851Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n" time=2026-09-22T21:50:04.426Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j4661888-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:westus3 role:hcp-underlay-ci01-j4661888-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:8df94b5e-187a-4556-af6a-9b474ecc0b97 region:westus3 secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#8df94b5e-187a-4556-af6a-9b474ecc0b97 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]" time=2026-09-22T21:50:04.426Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T21:50:04.653Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit time=2026-09-22T21:50:04.887Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false time=2026-09-22T21:50:04.887Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T21:50:05.009Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T22:00:08.502Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded" time=2026-09-22T22:00:08.503Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded" time=2026-09-22T22:01:08.503Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n" time=2026-09-22T22:01:09.036Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j4661888-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:westus3 role:hcp-underlay-ci01-j4661888-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:8df94b5e-187a-4556-af6a-9b474ecc0b97 region:westus3 secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#8df94b5e-187a-4556-af6a-9b474ecc0b97 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]" time=2026-09-22T22:01:09.036Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T22:01:09.387Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit time=2026-09-22T22:01:09.975Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T22:01:09.975Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T22:01:10.243Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1 time=2026-09-22T22:01:11.277Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T22:01:11.277Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder time=2026-09-22T22:01:11.584Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder time=2026-09-22T22:01:12.230Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder time=2026-09-22T22:01:13.220Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder time=2026-09-22T22:01:13.572Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder time=2026-09-22T22:01:14.205Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder time=2026-09-22T22:01:14.723Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder time=2026-09-22T22:01:15.282Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false time=2026-09-22T22:01:15.282Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T22:01:15.520Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1 time=2026-09-22T22:11:18.108Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded" time=2026-09-22T22:11:18.108Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded" time=2026-09-22T22:12:18.116Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n" time=2026-09-22T22:12:19.145Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j4661888-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:westus3 role:hcp-underlay-ci01-j4661888-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:8df94b5e-187a-4556-af6a-9b474ecc0b97 region:westus3 secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#8df94b5e-187a-4556-af6a-9b474ecc0b97 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]" time=2026-09-22T22:12:19.145Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T22:12:19.438Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit time=2026-09-22T22:12:19.970Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T22:12:19.970Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T22:12:20.187Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2 time=2026-09-22T22:12:21.247Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T22:12:21.247Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder time=2026-09-22T22:12:21.585Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder time=2026-09-22T22:12:22.170Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder time=2026-09-22T22:12:22.930Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder time=2026-09-22T22:12:23.284Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder time=2026-09-22T22:12:23.764Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder time=2026-09-22T22:12:24.305Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder time=2026-09-22T22:12:24.890Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false time=2026-09-22T22:12:24.890Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T22:12:25.135Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2 time=2026-09-22T22:20:03.973Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded" time=2026-09-22T22:20:03.973Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded" alert (3) alert [svc] ArobitForwarderJobUp fired 21:01:10 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ArobitForwarderJobUp does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T22:06:23Z Severity: Sev3 Labels: alertname="ArobitForwarderJobUp", cluster="ci01-j4661888-svc", component="monitoring-infrastructure", region="westus3", severity="warning" Description: The Arobit forwarder metrics endpoint on cluster ci01-j4661888-svc has been unreachable for 15 minutes. alert [svc] KubePodNotReady fired 21:01:10 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T21:56:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4661888-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-2hfrg", region="westus3", severity="warning" Description: Pod arobit/arobit-forwarder-2hfrg has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-09-22T21:56:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4661888-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-xc5hd", region="westus3", severity="warning" Description: Pod arobit/arobit-forwarder-xc5hd has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-09-22T21:56:20Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4661888-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-74xjw", region="westus3", severity="warning" Description: Pod arobit/arobit-forwarder-74xjw has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-09-22T21:56:21Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j4661888-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-cnj5s", region="westus3", severity="warning" Description: Pod arobit/arobit-forwarder-cnj5s has been in a non-ready state for longer than 5 minutes. alert [svc] KubeDaemonSetRolloutStuck fired 21:01:10 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubeDaemonSetRolloutStuck does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T22:06:23Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j4661888-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4661888/providers/microsoft.monitor/accounts/services-j4661888", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-lrr2t", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes. Firing 2: State: Fired (not resolved) Started: 2026-09-22T22:06:23Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j4661888-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.56:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4661888/providers/microsoft.monitor/accounts/services-j4661888", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-lrr2t", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
20:46:07	2h 6m 56s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head f79a2b4 · merge f79a2b4	westus3	alert	failed	★#7064 (merged)	1	alert (1) alert [svc] userJourneyFrontendLatency6h30m fired 20:46:07 UTC · failed at alert · test [aro-hcp-observability] [svc] alert userJourneyFrontendLatency6h30m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T22:28:47Z Severity: Sev3 Labels: alertname="userJourneyFrontendLatency6h30m", cluster="ci01-j9524736-svc", component="slo", long_window="6h", region="westus3", severity="3", short_window="30m", slo="frontend-latency" Description: More than 6% of Frontend requests on cluster ci01-j9524736-svc exceeded 1s on both the 6h and 30m windows (6x burn of the 1% latency budget).
20:39:21	1h 59m 2s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head cb250bc · merge cb250bc	westus3	n/a	passed	#7086 (merged)	0	n/a
20:20:22	2h 20m 33s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 0af6afb	centralus	alert, e2e	failed	#7084 (open)	3	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 20:20:22 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0015ac090>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-dbblg5cjpg9w" to finish deleting: GET https://rp.j9054592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0b7ee6c5-b14b-4d17-a809-95e9f3fb536a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0b7ee6c5-b14b-4d17-a809-95e9f3fb536a", "name": "0b7ee6c5-b14b-4d17-a809-95e9f3fb536a", "status": "Failed", "startTime": "2026-09-22T21:59:48.810888582Z", "endTime": "2026-09-22T22:23:48.20051154Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0a3mthe182nict37k3988rs7tsgdm3 still exists (deletion dispatched at 2026-09-22T21:59:51Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0015ac090>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-dbblg5cjpg9w" to finish deleting: GET https://rp.j9054592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0b7ee6c5-b14b-4d17-a809-95e9f3fb536a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0b7ee6c5-b14b-4d17-a809-95e9f3fb536a", "name": "0b7ee6c5-b14b-4d17-a809-95e9f3fb536a", "status": "Failed", "startTime": "2026-09-22T21:59:48.810888582Z", "endTime": "2026-09-22T22:23:48.20051154Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0a3mthe182nict37k3988rs7tsgdm3 still exists (deletion dispatched at 2026-09-22T21:59:51Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [svc] FrontendPathMedianLatency fired 20:20:22 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T21:51:11Z Ended: 2026-09-22T21:57:11Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j9054592-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9054592-svc). alert [svc] FrontendPathLatency fired 20:20:22 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T21:42:08Z Ended: 2026-09-22T22:16:09Z Severity: Sev3 Labels: alertname="FrontendPathLatency", cluster="ci01-j9054592-svc", component="frontend", method="get", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning" Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j9054592-svc).
20:17:39	2h 13m 58s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 7037eb8	canadacentral	alert, e2e	failed	#6661 (open)	2	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 20:17:39 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00250e0a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-t2kdhs" in resourcegroup="rg-np-version-upgrade-t2kdhs-5vn4clcwghqp" to finish deleting: GET https://rp.j5645440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/31e9dfbb-4e6b-4928-875b-adfc122ac940 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/31e9dfbb-4e6b-4928-875b-adfc122ac940", "name": "31e9dfbb-4e6b-4928-875b-adfc122ac940", "status": "Failed", "startTime": "2026-09-22T21:50:57.166246559Z", "endTime": "2026-09-22T22:14:57.071797622Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0a2vdu59ka2it7qevdj75a5me0ng8e still exists (deletion dispatched at 2026-09-22T21:50:58Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00250e0a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-t2kdhs" in resourcegroup="rg-np-version-upgrade-t2kdhs-5vn4clcwghqp" to finish deleting: GET https://rp.j5645440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/31e9dfbb-4e6b-4928-875b-adfc122ac940 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/31e9dfbb-4e6b-4928-875b-adfc122ac940", "name": "31e9dfbb-4e6b-4928-875b-adfc122ac940", "status": "Failed", "startTime": "2026-09-22T21:50:57.166246559Z", "endTime": "2026-09-22T22:14:57.071797622Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0a2vdu59ka2it7qevdj75a5me0ng8e still exists (deletion dispatched at 2026-09-22T21:50:58Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] FrontendPathMedianLatency fired 20:17:39 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T21:41:14Z Ended: 2026-09-22T22:20:13Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j5645440-svc", component="frontend", method="delete", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for delete /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j5645440-svc).
20:00:29	2h 51m 35s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 115a341	westus3	e2e	failed	#6821 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 20:00:29 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000f760a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-21-49lxzl" in resourcegroup="rg-cp-ystream-upgrade-4-21-ks5lgvsbcxp8" to finish deleting: GET https://rp.j8732288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e172962-dab6-4e45-8d0d-3db19b278332 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e172962-dab6-4e45-8d0d-3db19b278332", "name": "5e172962-dab6-4e45-8d0d-3db19b278332", "status": "Failed", "startTime": "2026-09-22T22:02:28.107089842Z", "endTime": "2026-09-22T22:26:33.710677175Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0ad9gsjkb0dmt6ds8umuk7neh3i2c8 still exists (deletion dispatched at 2026-09-22T22:02:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000f760a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-21-49lxzl" in resourcegroup="rg-cp-ystream-upgrade-4-21-ks5lgvsbcxp8" to finish deleting: GET https://rp.j8732288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e172962-dab6-4e45-8d0d-3db19b278332 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e172962-dab6-4e45-8d0d-3db19b278332", "name": "5e172962-dab6-4e45-8d0d-3db19b278332", "status": "Failed", "startTime": "2026-09-22T22:02:28.107089842Z", "endTime": "2026-09-22T22:26:33.710677175Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0ad9gsjkb0dmt6ds8umuk7neh3i2c8 still exists (deletion dispatched at 2026-09-22T22:02:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
19:29:34	3h 13m 54s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 54fa83b	canadacentral	alert, e2e	failed	#6993 (open)	2	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 19:29:34 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00152c108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-drcfdlgkcfg5" to finish deleting: GET https://rp.j9327488.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a61bbf5a-6f6d-430a-a670-57c9accc9081 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a61bbf5a-6f6d-430a-a670-57c9accc9081", "name": "a61bbf5a-6f6d-430a-a670-57c9accc9081", "status": "Failed", "startTime": "2026-09-22T22:01:22.621456872Z", "endTime": "2026-09-22T22:25:28.233887055Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0a4erurouv88b1rf2h4lagi5f4egc1 still exists (deletion dispatched at 2026-09-22T22:01:23Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00152c108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-drcfdlgkcfg5" to finish deleting: GET https://rp.j9327488.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a61bbf5a-6f6d-430a-a670-57c9accc9081 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a61bbf5a-6f6d-430a-a670-57c9accc9081", "name": "a61bbf5a-6f6d-430a-a670-57c9accc9081", "status": "Failed", "startTime": "2026-09-22T22:01:22.621456872Z", "endTime": "2026-09-22T22:25:28.233887055Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0a4erurouv88b1rf2h4lagi5f4egc1 still exists (deletion dispatched at 2026-09-22T22:01:23Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] ServiceMemoryDrift fired 19:29:34 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T22:23:58Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j9327488-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-9ld5r", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-9ld5r (namespace arobit) on cluster ci01-j9327488-svc has averaged 126.6% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
19:15:19	2h 8m 2s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 9ef5982	centralus	e2e	failed	#6925 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 19:15:19 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001c52240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-cn8tj9" in resourcegroup="rg-np-version-upgrade-cn8tj9-htwhhv8x7nxj" to finish deleting: GET https://rp.j2357248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/65d5b5af-f827-49a2-b835-b18ea96d28af -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/65d5b5af-f827-49a2-b835-b18ea96d28af", "name": "65d5b5af-f827-49a2-b835-b18ea96d28af", "status": "Failed", "startTime": "2026-09-22T20:42:52.522338422Z", "endTime": "2026-09-22T21:06:55.676641569Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t093k6i18qur3n9jj565dmnt62jj5aq still exists (deletion dispatched at 2026-09-22T20:42:54Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001c52240>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-cn8tj9" in resourcegroup="rg-np-version-upgrade-cn8tj9-htwhhv8x7nxj" to finish deleting: GET https://rp.j2357248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/65d5b5af-f827-49a2-b835-b18ea96d28af -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/65d5b5af-f827-49a2-b835-b18ea96d28af", "name": "65d5b5af-f827-49a2-b835-b18ea96d28af", "status": "Failed", "startTime": "2026-09-22T20:42:52.522338422Z", "endTime": "2026-09-22T21:06:55.676641569Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t093k6i18qur3n9jj565dmnt62jj5aq still exists (deletion dispatched at 2026-09-22T20:42:54Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
19:02:04	3h 30m 31s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 8ae5d54 · merge 8ae5d54	canadacentral	n/a	passed	#7067 (merged)	0	n/a
18:22:13	2h 2m 27s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c99d95a	centralus	alert, provision	failed	#6820 (open)	4	provision (1) failed to roll out Helm release: context deadline exceeded 18:22:13 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/arobit · suite templatize-pipeline Show raw failure time=2026-09-22T19:48:43.126Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n" time=2026-09-22T19:48:43.619Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j3875072-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:centralus role:hcp-underlay-ci01-j3875072-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:f4808719-7718-4dd5-a60e-1d3b526f1087 region:centralus secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#f4808719-7718-4dd5-a60e-1d3b526f1087 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]" time=2026-09-22T19:48:43.619Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T19:48:43.778Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit time=2026-09-22T19:48:43.960Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false time=2026-09-22T19:48:43.960Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T19:48:44.045Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T19:58:46.492Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded" time=2026-09-22T19:58:46.492Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded" time=2026-09-22T19:59:46.495Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n" time=2026-09-22T19:59:47.378Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j3875072-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:centralus role:hcp-underlay-ci01-j3875072-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:f4808719-7718-4dd5-a60e-1d3b526f1087 region:centralus secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#f4808719-7718-4dd5-a60e-1d3b526f1087 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]" time=2026-09-22T19:59:47.379Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T19:59:47.630Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit time=2026-09-22T19:59:48.103Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T19:59:48.103Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T19:59:48.332Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1 time=2026-09-22T19:59:49.351Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T19:59:49.352Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder time=2026-09-22T19:59:49.594Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder time=2026-09-22T19:59:49.976Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder time=2026-09-22T19:59:50.450Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder time=2026-09-22T19:59:50.618Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder time=2026-09-22T19:59:50.774Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder time=2026-09-22T19:59:51.169Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder time=2026-09-22T19:59:51.518Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false time=2026-09-22T19:59:51.518Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T19:59:51.622Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1 time=2026-09-22T20:09:53.731Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded" time=2026-09-22T20:09:53.731Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded" time=2026-09-22T20:10:53.735Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n" time=2026-09-22T20:10:54.409Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j3875072-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:centralus role:hcp-underlay-ci01-j3875072-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:f4808719-7718-4dd5-a60e-1d3b526f1087 region:centralus secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#f4808719-7718-4dd5-a60e-1d3b526f1087 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]" time=2026-09-22T20:10:54.409Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T20:10:54.723Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit time=2026-09-22T20:10:55.205Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T20:10:55.205Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T20:10:55.345Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2 time=2026-09-22T20:10:56.191Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T20:10:56.191Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder time=2026-09-22T20:10:56.565Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder time=2026-09-22T20:10:56.839Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder time=2026-09-22T20:10:57.169Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder time=2026-09-22T20:10:57.365Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder time=2026-09-22T20:10:57.525Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder time=2026-09-22T20:10:57.721Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder time=2026-09-22T20:10:58.033Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false time=2026-09-22T20:10:58.033Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit time=2026-09-22T20:10:58.139Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2 time=2026-09-22T20:18:43.221Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded" time=2026-09-22T20:18:43.222Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded" alert (3) alert [svc] KubeDaemonSetRolloutStuck fired 18:22:13 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubeDaemonSetRolloutStuck does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T20:05:10Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j3875072-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.70:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3875072/providers/microsoft.monitor/accounts/services-j3875072", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-lbmz5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes. Firing 2: State: Fired (not resolved) Started: 2026-09-22T20:05:10Z Severity: Sev3 Labels: alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j3875072-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.70:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3875072/providers/microsoft.monitor/accounts/services-j3875072", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-lbmz5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes. alert [svc] ArobitForwarderJobUp fired 18:22:13 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ArobitForwarderJobUp does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T20:05:27Z Severity: Sev3 Labels: alertname="ArobitForwarderJobUp", cluster="ci01-j3875072-svc", component="monitoring-infrastructure", region="centralus", severity="warning" Description: The Arobit forwarder metrics endpoint on cluster ci01-j3875072-svc has been unreachable for 15 minutes. alert [svc] KubePodNotReady fired 18:22:13 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 5 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T19:55:09Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3875072-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-8sx8x", region="centralus", severity="warning" Description: Pod arobit/arobit-forwarder-8sx8x has been in a non-ready state for longer than 5 minutes. Firing 2: State: Fired (not resolved) Started: 2026-09-22T19:55:09Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3875072-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-zpdt2", region="centralus", severity="warning" Description: Pod arobit/arobit-forwarder-zpdt2 has been in a non-ready state for longer than 5 minutes. Firing 3: State: Fired (not resolved) Started: 2026-09-22T19:55:09Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3875072-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-pc755", region="centralus", severity="warning" Description: Pod arobit/arobit-forwarder-pc755 has been in a non-ready state for longer than 5 minutes. Firing 4: State: Fired (not resolved) Started: 2026-09-22T19:55:09Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3875072-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-kzj4q", region="centralus", severity="warning" Description: Pod arobit/arobit-forwarder-kzj4q has been in a non-ready state for longer than 5 minutes. Firing 5: State: Fired (not resolved) Started: 2026-09-22T19:57:08Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j3875072-mgmt-2", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-749558849-6khd4", region="centralus", severity="warning" Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-749558849-6khd4 has been in a non-ready state for longer than 5 minutes.
18:08:24	2h 12m 14s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head d433c21 · merge d433c21	westus3	n/a	passed	#6752 (merged)	0	n/a
18:00:43	2h 41m 39s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head e8df860 · merge e8df860	westus3	alert, e2e	failed	#5846 (merged)	4	e2e (3) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 18:00:43 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00084a288>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-68rjtkpsqm7b" to finish deleting: GET https://rp.j0702464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6084bb0c-7df8-4842-ab2e-b53a4d94db1d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6084bb0c-7df8-4842-ab2e-b53a4d94db1d", "name": "6084bb0c-7df8-4842-ab2e-b53a4d94db1d", "status": "Failed", "startTime": "2026-09-22T19:42:12.055385484Z", "endTime": "2026-09-22T20:06:15.982381047Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t089ppdctio8jds461l0pe41vnps034 still exists (deletion dispatched at 2026-09-22T19:42:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00084a288>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-68rjtkpsqm7b" to finish deleting: GET https://rp.j0702464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6084bb0c-7df8-4842-ab2e-b53a4d94db1d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6084bb0c-7df8-4842-ab2e-b53a4d94db1d", "name": "6084bb0c-7df8-4842-ab2e-b53a4d94db1d", "status": "Failed", "startTime": "2026-09-22T19:42:12.055385484Z", "endTime": "2026-09-22T20:06:15.982381047Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t089ppdctio8jds461l0pe41vnps034 still exists (deletion dispatched at 2026-09-22T19:42:13Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist 18:00:43 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.002s. Certificate should be issued by Let's Encrypt staging via ACME DNS-01 Expected success, but got an error: <errors.errorString \| 0xc002455530>: certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.002s. Certificate should be issued by Let's Encrypt staging via ACME DNS-01 Expected success, but got an error: <errors.errorString \| 0xc002455530>: certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist ... ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 18:00:43 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0017a8228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-jq6hhtmkbzt4" to finish deleting: GET https://rp.j0702464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff92afaf-c9d4-443d-b74a-95c2f3e15d15 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff92afaf-c9d4-443d-b74a-95c2f3e15d15", "name": "ff92afaf-c9d4-443d-b74a-95c2f3e15d15", "status": "Failed", "startTime": "2026-09-22T20:01:15.219491093Z", "endTime": "2026-09-22T20:25:19.472856279Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t089usm9gvf5gahfe2druepcei6i824 still exists (deletion dispatched at 2026-09-22T20:01:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0017a8228>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-jq6hhtmkbzt4" to finish deleting: GET https://rp.j0702464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff92afaf-c9d4-443d-b74a-95c2f3e15d15 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ff92afaf-c9d4-443d-b74a-95c2f3e15d15", "name": "ff92afaf-c9d4-443d-b74a-95c2f3e15d15", "status": "Failed", "startTime": "2026-09-22T20:01:15.219491093Z", "endTime": "2026-09-22T20:25:19.472856279Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t089usm9gvf5gahfe2druepcei6i824 still exists (deletion dispatched at 2026-09-22T20:01:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] ServiceMemoryDrift fired 18:00:43 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T20:26:03Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j0702464-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-7j8m6", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-7j8m6 (namespace arobit) on cluster ci01-j0702464-svc has averaged 125.4% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
17:48:09	2h 37m 37s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 6cb0492 · merge 6cb0492	centralus	alert, e2e	failed	#7020 (merged)	4	e2e (2) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 17:48:09 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0011083d8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-skip-minor-zzfkr9" in resourcegroup="rg-np-skip-minor-zzfkr9-gb95nn9vpwbx" to finish deleting: GET https://rp.j8706944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8986523a-9554-40d0-9d12-c9d8f53ca3b2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8986523a-9554-40d0-9d12-c9d8f53ca3b2", "name": "8986523a-9554-40d0-9d12-c9d8f53ca3b2", "status": "Failed", "startTime": "2026-09-22T19:40:48.963450447Z", "endTime": "2026-09-22T20:04:49.182718803Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t082ht679tj6q9hdfrhosuc2f437sef still exists (deletion dispatched at 2026-09-22T19:40:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0011083d8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-skip-minor-zzfkr9" in resourcegroup="rg-np-skip-minor-zzfkr9-gb95nn9vpwbx" to finish deleting: GET https://rp.j8706944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8986523a-9554-40d0-9d12-c9d8f53ca3b2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8986523a-9554-40d0-9d12-c9d8f53ca3b2", "name": "8986523a-9554-40d0-9d12-c9d8f53ca3b2", "status": "Failed", "startTime": "2026-09-22T19:40:48.963450447Z", "endTime": "2026-09-22T20:04:49.182718803Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t082ht679tj6q9hdfrhosuc2f437sef still exists (deletion dispatched at 2026-09-22T19:40:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 17:48:09 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0011de180>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-wgqfmcc7mttj" to finish deleting: GET https://rp.j8706944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a537a987-ddb8-49ee-9e92-79fbe69e65a6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a537a987-ddb8-49ee-9e92-79fbe69e65a6", "name": "a537a987-ddb8-49ee-9e92-79fbe69e65a6", "status": "Failed", "startTime": "2026-09-22T19:44:00.645194734Z", "endTime": "2026-09-22T20:08:09.172667585Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t082ge25c3lks1mu3j5dtlgjr539r8q still exists (deletion dispatched at 2026-09-22T19:44:04Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0011de180>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-wgqfmcc7mttj" to finish deleting: GET https://rp.j8706944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a537a987-ddb8-49ee-9e92-79fbe69e65a6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a537a987-ddb8-49ee-9e92-79fbe69e65a6", "name": "a537a987-ddb8-49ee-9e92-79fbe69e65a6", "status": "Failed", "startTime": "2026-09-22T19:44:00.645194734Z", "endTime": "2026-09-22T20:08:09.172667585Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t082ge25c3lks1mu3j5dtlgjr539r8q still exists (deletion dispatched at 2026-09-22T19:44:04Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [svc] ServiceMemoryDrift fired 17:48:09 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T20:12:36Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j8706944-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-5w4zq", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-5w4zq (namespace arobit) on cluster ci01-j8706944-svc has averaged 125.4% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml. alert [svc] FrontendPathMedianLatency fired 17:48:09 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T19:32:14Z Ended: 2026-09-22T19:38:14Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j8706944-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j8706944-svc).
17:22:16	39m 33s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head d433c21 · merge d433c21	n/a	unknown	failed	#6752 (merged)	1	non-artifact-backed failure (no junit artifacts)
17:15:09	2h 24m 12s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 244d0c1 · merge 244d0c1	canadacentral	alert, e2e	failed	#4878 (merged)	4	e2e (2) DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded 17:15:09 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:188]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-zck558" Unexpected error: <fmt.wrapError \| 0xc001112760>: DNS for route host agnhost-e2e-sample-app-wrj85.apps.aro.h6n7n8r8p3m8n7w.1qn2.j8935808.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-wrj85.apps.aro.h6n7n8r8p3m8n7w.1qn2.j8935808.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wrj85.apps.aro.h6n7n8r8p3m8n7w.1qn2.j8935808.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:188]: failed to verify simple web app runs on cluster "cluster-candidate-5-0-zck558" Unexpected error: <fmt.wrapError \| 0xc001112760>: DNS for route host agnhost-e2e-sample-app-wrj85.apps.aro.h6n7n8r8p3m8n7w.1qn2.j8935808.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-wrj85.apps.aro.h6n7n8r8p3m8n7w.1qn2.j8935808.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-wrj85.apps.aro.h6n7n8r8p3m8n7w.1qn2.j8935808.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 17:15:09 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00107a1e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-sfwkw6t6m866" to finish deleting: GET https://rp.j8935808.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2c2bbf72-d927-4c9c-b696-4f1f281075fe -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2c2bbf72-d927-4c9c-b696-4f1f281075fe", "name": "2c2bbf72-d927-4c9c-b696-4f1f281075fe", "status": "Failed", "startTime": "2026-09-22T18:57:36.550691584Z", "endTime": "2026-09-22T19:21:39.730536375Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t07dfina4snjdhqqt1e8v0e6qcnbvh6 still exists (deletion dispatched at 2026-09-22T18:57:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00107a1e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-sfwkw6t6m866" to finish deleting: GET https://rp.j8935808.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2c2bbf72-d927-4c9c-b696-4f1f281075fe -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2c2bbf72-d927-4c9c-b696-4f1f281075fe", "name": "2c2bbf72-d927-4c9c-b696-4f1f281075fe", "status": "Failed", "startTime": "2026-09-22T18:57:36.550691584Z", "endTime": "2026-09-22T19:21:39.730536375Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t07dfina4snjdhqqt1e8v0e6qcnbvh6 still exists (deletion dispatched at 2026-09-22T18:57:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [svc] FrontendPathMedianLatency fired 17:15:09 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T18:50:27Z Ended: 2026-09-22T19:06:27Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j8935808-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters", severity="warning" Description: The median (p50) of frontend request latency for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters has exceeded 250ms over the past 30 minutes (cluster ci01-j8935808-svc). alert [svc] ServiceMemoryDrift fired 17:15:09 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T19:25:37Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j8935808-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-mfjw9", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-mfjw9 (namespace arobit) on cluster ci01-j8935808-svc has averaged 127.2% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
17:01:22	2h 0m 49s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 115a341	westus3	alert, e2e	failed	#6821 (open)	4	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 17:01:22 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00124a138>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-rztn7tgf9xgj" to finish deleting: GET https://rp.j7946880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2caaa78a-cc90-4669-8d96-1a0204de9f98 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2caaa78a-cc90-4669-8d96-1a0204de9f98", "name": "2caaa78a-cc90-4669-8d96-1a0204de9f98", "status": "Failed", "startTime": "2026-09-22T18:21:26.160080988Z", "endTime": "2026-09-22T18:45:29.126992957Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t06sdvhi784tdu6s15hf9hcnp7dpebk still exists (deletion dispatched at 2026-09-22T18:21:28Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00124a138>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-rztn7tgf9xgj" to finish deleting: GET https://rp.j7946880.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2caaa78a-cc90-4669-8d96-1a0204de9f98 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2caaa78a-cc90-4669-8d96-1a0204de9f98", "name": "2caaa78a-cc90-4669-8d96-1a0204de9f98", "status": "Failed", "startTime": "2026-09-22T18:21:26.160080988Z", "endTime": "2026-09-22T18:45:29.126992957Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t06sdvhi784tdu6s15hf9hcnp7dpebk still exists (deletion dispatched at 2026-09-22T18:21:28Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (3) alert [svc] FrontendPathMedianLatency fired 17:01:22 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T17:42:06Z Ended: 2026-09-22T18:17:10Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j7946880-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j7946880-svc). alert [svc] FrontendPathLatency fired 17:01:22 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T18:19:06Z Ended: 2026-09-22T18:28:06Z Severity: Sev3 Labels: alertname="FrontendPathLatency", cluster="ci01-j7946880-svc", component="frontend", method="delete", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: More than 5% of frontend requests for delete /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j7946880-svc). alert [svc] ServiceMemoryDrift fired 17:01:22 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T18:47:34Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j7946880-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-674px", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-674px (namespace arobit) on cluster ci01-j7946880-svc has averaged 127.1% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
17:01:00	1h 44m 38s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head a66c056 · merge a66c056	canadacentral	n/a	passed	#7079 (merged)	0	n/a
16:36:12	2h 4m 45s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head e82abf6	westus3	n/a	passed	#6858 (open)	0	n/a
16:02:36	2h 8m 34s	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 6c5cfd0 · merge a0bf660	westus3	alert, e2e	failed	#6503 (merged)	5	e2e (2) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 16:02:36 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-gp68bk" with version 4.20.38 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00011cba0>: failed to create HCP cluster cluster-zstream-4-20-gp68bk: failed waiting for cluster="cluster-zstream-4-20-gp68bk" in resourcegroup="rg-zstream-upgrade-4-20-qcsqzkhckgg6" to finish creating: GET https://rp.j0115840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65ff5f37-d01d-4885-92e1-a54bf276ecdf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65ff5f37-d01d-4885-92e1-a54bf276ecdf", "name": "65ff5f37-d01d-4885-92e1-a54bf276ecdf", "status": "Failed", "startTime": "2026-09-22T16:56:56.107653498Z", "endTime": "2026-09-22T17:26:04.415591039Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-gp68bk" with version 4.20.38 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00011cba0>: failed to create HCP cluster cluster-zstream-4-20-gp68bk: failed waiting for cluster="cluster-zstream-4-20-gp68bk" in resourcegroup="rg-zstream-upgrade-4-20-qcsqzkhckgg6" to finish creating: GET https://rp.j0115840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65ff5f37-d01d-4885-92e1-a54bf276ecdf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/65ff5f37-d01d-4885-92e1-a54bf276ecdf", "name": "65ff5f37-d01d-4885-92e1-a54bf276ecdf", "status": "Failed", "startTime": "2026-09-22T16:56:56.107653498Z", "endTime": "2026-09-22T17:26:04.415591039Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred at least one schedule should have a LastBackupTime 16:02:36 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.007s. at least one schedule should have a LastBackupTime Expected <bool>: ... to be true fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.007s. at least one schedule should have a LastBackupTime Expected <bool>: ... to be true alert (3) alert [svc] KubeDeploymentReplicasMismatch fired 16:02:36 UTC · failed at alert · test [aro-hcp-observability] [svc] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-09-22T17:30:47Z Ended: 2026-09-22T17:38:47Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0115840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0115840/providers/microsoft.monitor/accounts/services-j0115840", namespace="klusterlet-2t06amajdmngpo94r6274auvf1ibbeql", pod="arohcp-monitor-kube-state-metrics-7955558448-mpcb7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2t06amajdmngpo94r6274auvf1ibbeql/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-09-22T17:30:47Z Ended: 2026-09-22T17:37:47Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j0115840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0115840/providers/microsoft.monitor/accounts/services-j0115840", namespace="klusterlet-2t06amajdmngpo94r6274auvf1ibbeql", pod="arohcp-monitor-kube-state-metrics-7955558448-mpcb7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment klusterlet-2t06amajdmngpo94r6274auvf1ibbeql/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes. alert [hcp] KubeStatefulSetReplicasMismatch fired 16:02:36 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-09-22T17:17:06Z Ended: 2026-09-22T17:45:02Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0115840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0115840/providers/microsoft.monitor/accounts/hcps-j0115840", namespace="ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i", pod="arohcp-monitor-kube-state-metrics-7955558448-mpcb7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-09-22T17:17:06Z Ended: 2026-09-22T17:49:03Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0115840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0115840/providers/microsoft.monitor/accounts/hcps-j0115840", namespace="ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i", pod="arohcp-monitor-kube-state-metrics-7955558448-mpcb7", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i/etcd has not matched the expected number of replicas for longer than 15 minutes. alert [hcp] KubePodCrashLooping fired 16:02:36 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-09-22T17:19:59Z Ended: 2026-09-22T17:44:00Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0115840-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0115840/providers/microsoft.monitor/accounts/hcps-j0115840", namespace="ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="603dfbeb-4dca-4193-8523-f1040f307d47" Description: Pod ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff"). Firing 2: State: Resolved Started: 2026-09-22T17:19:59Z Ended: 2026-09-22T17:48:59Z Severity: Sev3 Labels: alertname="KubePodCrashLooping", cluster="ci01-j0115840-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0115840/providers/microsoft.monitor/accounts/hcps-j0115840", namespace="ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="603dfbeb-4dca-4193-8523-f1040f307d47" Description: Pod ocm-arohcpci01-2t06amajdmngpo94r6274auvf1ibbeql-x5r1r1c7h5x7z3i/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").
16:02:20	2h 5m 5s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 8ae5d54 · merge 8ae5d54	canadacentral	e2e	failed	#7067 (merged)	1	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 16:02:20 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0011c40f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-m9nlwn" in resourcegroup="rg-cp-ystream-upgrade-4-22-cwk2zbwbl8df" to finish deleting: GET https://rp.j5202816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e162fce1-69ff-47e2-800c-9e9a7da178b9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e162fce1-69ff-47e2-800c-9e9a7da178b9", "name": "e162fce1-69ff-47e2-800c-9e9a7da178b9", "status": "Failed", "startTime": "2026-09-22T17:20:42.567454265Z", "endTime": "2026-09-22T17:44:47.318967939Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0673hl1rs5ia5vgvj3arfsusevukrr still exists (deletion dispatched at 2026-09-22T17:20:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0011c40f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-m9nlwn" in resourcegroup="rg-cp-ystream-upgrade-4-22-cwk2zbwbl8df" to finish deleting: GET https://rp.j5202816.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e162fce1-69ff-47e2-800c-9e9a7da178b9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e162fce1-69ff-47e2-800c-9e9a7da178b9", "name": "e162fce1-69ff-47e2-800c-9e9a7da178b9", "status": "Failed", "startTime": "2026-09-22T17:20:42.567454265Z", "endTime": "2026-09-22T17:44:47.318967939Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0673hl1rs5ia5vgvj3arfsusevukrr still exists (deletion dispatched at 2026-09-22T17:20:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
15:50:24	2h 10m 18s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 610c52d	centralus	e2e	failed	#6661 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:50:24 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001718168>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-sr9mdkjlp5d6" to finish deleting: GET https://rp.j4643968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/db9563c3-fa41-4db8-95de-b8fb6d868146 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/db9563c3-fa41-4db8-95de-b8fb6d868146", "name": "db9563c3-fa41-4db8-95de-b8fb6d868146", "status": "Failed", "startTime": "2026-09-22T17:18:42.990998968Z", "endTime": "2026-09-22T17:42:42.450346964Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t05uv4e9fgg9ln37cnv09e5j15gkrtl still exists (deletion dispatched at 2026-09-22T17:18:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001718168>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-sr9mdkjlp5d6" to finish deleting: GET https://rp.j4643968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/db9563c3-fa41-4db8-95de-b8fb6d868146 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/db9563c3-fa41-4db8-95de-b8fb6d868146", "name": "db9563c3-fa41-4db8-95de-b8fb6d868146", "status": "Failed", "startTime": "2026-09-22T17:18:42.990998968Z", "endTime": "2026-09-22T17:42:42.450346964Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t05uv4e9fgg9ln37cnv09e5j15gkrtl still exists (deletion dispatched at 2026-09-22T17:18:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
15:46:49	2h 18m 45s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 3e95ed9	westus3	alert, e2e	failed	#6900 (open)	3	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:46:49 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000f30168>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-dpkm5d989phc" to finish deleting: GET https://rp.j3599104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5496eb36-fed0-418b-b4a4-1bbb730977c5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5496eb36-fed0-418b-b4a4-1bbb730977c5", "name": "5496eb36-fed0-418b-b4a4-1bbb730977c5", "status": "Failed", "startTime": "2026-09-22T17:23:06.139309331Z", "endTime": "2026-09-22T17:47:12.014996899Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t061bpco7v1l3lgu542dvncm4grjrq2 still exists (deletion dispatched at 2026-09-22T17:23:07Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000f30168>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-dpkm5d989phc" to finish deleting: GET https://rp.j3599104.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5496eb36-fed0-418b-b4a4-1bbb730977c5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5496eb36-fed0-418b-b4a4-1bbb730977c5", "name": "5496eb36-fed0-418b-b4a4-1bbb730977c5", "status": "Failed", "startTime": "2026-09-22T17:23:06.139309331Z", "endTime": "2026-09-22T17:47:12.014996899Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t061bpco7v1l3lgu542dvncm4grjrq2 still exists (deletion dispatched at 2026-09-22T17:23:07Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [svc] BackendControllerQueueDepthHigh fired 15:46:49 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T16:52:14Z Ended: 2026-09-22T16:58:11Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3599104-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning" Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert [svc] ServiceMemoryDrift fired 15:46:49 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T17:50:10Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j3599104-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-79p2m", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-79p2m (namespace arobit) on cluster ci01-j3599104-svc has averaged 124.6% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
15:46:19	2h 14m 22s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head f82618a	westus3	alert, e2e	failed	⚠#6889 (open)	63	e2e (59) ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc00088c7a0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-6t86q5nsgrh9" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/315fcf5a-f184-4c52-a71c-88102dbc1533 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/315fcf5a-f184-4c52-a71c-88102dbc1533", "name": "315fcf5a-f184-4c52-a71c-88102dbc1533", "status": "Failed", "startTime": "2026-09-22T16:36:26.20084073Z", "endTime": "2026-09-22T17:05:34.327315045Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc00088c7a0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-6t86q5nsgrh9" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/315fcf5a-f184-4c52-a71c-88102dbc1533 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/315fcf5a-f184-4c52-a71c-88102dbc1533", "name": "315fcf5a-f184-4c52-a71c-88102dbc1533", "status": "Failed", "startTime": "2026-09-22T16:36:26.20084073Z", "endTime": "2026-09-22T17:05:34.327315045Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000f62760>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-fbdrrsb4f4qb" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/673f7571-13b3-4bc0-b005-ad87a3d3f04c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/673f7571-13b3-4bc0-b005-ad87a3d3f04c", "name": "673f7571-13b3-4bc0-b005-ad87a3d3f04c", "status": "Failed", "startTime": "2026-09-22T16:35:51.009988979Z", "endTime": "2026-09-22T17:04:54.001140954Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000f62760>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-fbdrrsb4f4qb" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/673f7571-13b3-4bc0-b005-ad87a3d3f04c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/673f7571-13b3-4bc0-b005-ad87a3d3f04c", "name": "673f7571-13b3-4bc0-b005-ad87a3d3f04c", "status": "Failed", "startTime": "2026-09-22T16:35:51.009988979Z", "endTime": "2026-09-22T17:04:54.001140954Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test SRE should return 409 when boot diagnostics is disabled on a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test Unexpected error: <fmt.wrapError \| 0xc000d06700>: failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-b5tsprkj7fb4" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20e6f9e0-2e0c-4d2d-81f1-3b3a636ac461 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20e6f9e0-2e0c-4d2d-81f1-3b3a636ac461", "name": "20e6f9e0-2e0c-4d2d-81f1-3b3a636ac461", "status": "Failed", "startTime": "2026-09-22T16:36:09.86553733Z", "endTime": "2026-09-22T17:05:14.887023579Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test Unexpected error: <fmt.wrapError \| 0xc000d06700>: failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-b5tsprkj7fb4" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20e6f9e0-2e0c-4d2d-81f1-3b3a636ac461 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20e6f9e0-2e0c-4d2d-81f1-3b3a636ac461", "name": "20e6f9e0-2e0c-4d2d-81f1-3b3a636ac461", "status": "Failed", "startTime": "2026-09-22T16:36:09.86553733Z", "endTime": "2026-09-22T17:05:14.887023579Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to retrieve cluster and nodepool status active versions · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:88]: failed to create HCP cluster "v20261001" via v20261001preview Unexpected error: <fmt.wrapError \| 0xc0004f49c0>: failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-tq6grpmch9pl" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9ef2ab64-6f5a-4ada-b76a-08072358a3a6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9ef2ab64-6f5a-4ada-b76a-08072358a3a6", "name": "9ef2ab64-6f5a-4ada-b76a-08072358a3a6", "status": "Failed", "startTime": "2026-09-22T16:35:50.115809759Z", "endTime": "2026-09-22T17:04:54.124624103Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:88]: failed to create HCP cluster "v20261001" via v20261001preview Unexpected error: <fmt.wrapError \| 0xc0004f49c0>: failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-tq6grpmch9pl" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9ef2ab64-6f5a-4ada-b76a-08072358a3a6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9ef2ab64-6f5a-4ada-b76a-08072358a3a6", "name": "9ef2ab64-6f5a-4ada-b76a-08072358a3a6", "status": "Failed", "startTime": "2026-09-22T16:35:50.115809759Z", "endTime": "2026-09-22T17:04:54.124624103Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc00110caa0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-z8xc4gxhmf84" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/12bbf6a8-0927-462a-952a-3bd89cfcdbb5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/12bbf6a8-0927-462a-952a-3bd89cfcdbb5", "name": "12bbf6a8-0927-462a-952a-3bd89cfcdbb5", "status": "Failed", "startTime": "2026-09-22T17:13:22.791211135Z", "endTime": "2026-09-22T17:42:28.16969575Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc00110caa0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-z8xc4gxhmf84" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/12bbf6a8-0927-462a-952a-3bd89cfcdbb5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/12bbf6a8-0927-462a-952a-3bd89cfcdbb5", "name": "12bbf6a8-0927-462a-952a-3bd89cfcdbb5", "status": "Failed", "startTime": "2026-09-22T17:13:22.791211135Z", "endTime": "2026-09-22T17:42:28.16969575Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes) Unexpected error: <exported.ResponseError \| 0xc00107a300>: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/82f81316-7913-421b-adf2-73a13e0c018f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/82f81316-7913-421b-adf2-73a13e0c018f", "name": "82f81316-7913-421b-adf2-73a13e0c018f", "status": "Failed", "startTime": "2026-09-22T16:37:13.699747469Z", "endTime": "2026-09-22T17:06:14.29231712Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes) Unexpected error: <exported.ResponseError \| 0xc00107a300>: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/82f81316-7913-421b-adf2-73a13e0c018f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/82f81316-7913-421b-adf2-73a13e0c018f", "name": "82f81316-7913-421b-adf2-73a13e0c018f", "status": "Failed", "startTime": "2026-09-22T16:37:13.699747469Z", "endTime": "2026-09-22T17:06:14.29231712Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-mqgx5p with version 4.21.34 Unexpected error: <fmt.wrapError \| 0xc0001a4ea0>: failed to create HCP cluster np-version-upgrade-cluster-mqgx5p: failed waiting for cluster="np-version-upgrade-cluster-mqgx5p" in resourcegroup="rg-np-version-upgrade-mqgx5p-sf4vhtxlsk5p" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b85ee66-6ecf-4aec-abd2-44ca10c2169c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b85ee66-6ecf-4aec-abd2-44ca10c2169c", "name": "7b85ee66-6ecf-4aec-abd2-44ca10c2169c", "status": "Failed", "startTime": "2026-09-22T16:36:28.335722772Z", "endTime": "2026-09-22T17:05:33.994048214Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-mqgx5p with version 4.21.34 Unexpected error: <fmt.wrapError \| 0xc0001a4ea0>: failed to create HCP cluster np-version-upgrade-cluster-mqgx5p: failed waiting for cluster="np-version-upgrade-cluster-mqgx5p" in resourcegroup="rg-np-version-upgrade-mqgx5p-sf4vhtxlsk5p" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b85ee66-6ecf-4aec-abd2-44ca10c2169c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b85ee66-6ecf-4aec-abd2-44ca10c2169c", "name": "7b85ee66-6ecf-4aec-abd2-44ca10c2169c", "status": "Failed", "startTime": "2026-09-22T16:36:28.335722772Z", "endTime": "2026-09-22T17:05:33.994048214Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test SRE should be able to retrieve serial console logs for a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test Unexpected error: <fmt.wrapError \| 0xc000f8cb00>: failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-v9tlgczmbzdv" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7ebec1a-d39e-4159-8ec0-c162b81238f9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7ebec1a-d39e-4159-8ec0-c162b81238f9", "name": "f7ebec1a-d39e-4159-8ec0-c162b81238f9", "status": "Failed", "startTime": "2026-09-22T16:36:11.943105645Z", "endTime": "2026-09-22T17:05:14.704022029Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test Unexpected error: <fmt.wrapError \| 0xc000f8cb00>: failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-v9tlgczmbzdv" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7ebec1a-d39e-4159-8ec0-c162b81238f9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7ebec1a-d39e-4159-8ec0-c162b81238f9", "name": "f7ebec1a-d39e-4159-8ec0-c162b81238f9", "status": "Failed", "startTime": "2026-09-22T16:36:11.943105645Z", "endTime": "2026-09-22T17:05:14.704022029Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000e4c800>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-vh5mlhlwh5jf" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/745705aa-7ec2-488a-b45f-576a0d31985e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/745705aa-7ec2-488a-b45f-576a0d31985e", "name": "745705aa-7ec2-488a-b45f-576a0d31985e", "status": "Failed", "startTime": "2026-09-22T16:36:19.073544004Z", "endTime": "2026-09-22T17:05:24.167779386Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000e4c800>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-vh5mlhlwh5jf" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/745705aa-7ec2-488a-b45f-576a0d31985e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/745705aa-7ec2-488a-b45f-576a0d31985e", "name": "745705aa-7ec2-488a-b45f-576a0d31985e", "status": "Failed", "startTime": "2026-09-22T16:36:19.073544004Z", "endTime": "2026-09-22T17:05:24.167779386Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc0011842a0>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-67zjc9d45r89" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9ef3226a-042d-43f6-9a4e-08f44f519818 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9ef3226a-042d-43f6-9a4e-08f44f519818", "name": "9ef3226a-042d-43f6-9a4e-08f44f519818", "status": "Failed", "startTime": "2026-09-22T16:36:12.003244297Z", "endTime": "2026-09-22T17:05:15.598887881Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc0011842a0>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-67zjc9d45r89" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9ef3226a-042d-43f6-9a4e-08f44f519818 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9ef3226a-042d-43f6-9a4e-08f44f519818", "name": "9ef3226a-042d-43f6-9a4e-08f44f519818", "status": "Failed", "startTime": "2026-09-22T16:36:12.003244297Z", "endTime": "2026-09-22T17:05:15.598887881Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-ntd64c Unexpected error: <fmt.wrapError \| 0xc000fa0ac0>: failed to create HCP cluster np-noedge-ntd64c: failed waiting for cluster="np-noedge-ntd64c" in resourcegroup="rg-np-noedge-ntd64c-g6pwtg6d4hs9" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8a13d95b-4e01-482c-9152-e5605fd1f2ba -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8a13d95b-4e01-482c-9152-e5605fd1f2ba", "name": "8a13d95b-4e01-482c-9152-e5605fd1f2ba", "status": "Failed", "startTime": "2026-09-22T16:36:15.242676605Z", "endTime": "2026-09-22T17:05:23.926389731Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-ntd64c Unexpected error: <fmt.wrapError \| 0xc000fa0ac0>: failed to create HCP cluster np-noedge-ntd64c: failed waiting for cluster="np-noedge-ntd64c" in resourcegroup="rg-np-noedge-ntd64c-g6pwtg6d4hs9" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8a13d95b-4e01-482c-9152-e5605fd1f2ba -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8a13d95b-4e01-482c-9152-e5605fd1f2ba", "name": "8a13d95b-4e01-482c-9152-e5605fd1f2ba", "status": "Failed", "startTime": "2026-09-22T16:36:15.242676605Z", "endTime": "2026-09-22T17:05:23.926389731Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred cluster entered terminal Failed state after role assignment deployment 15:46:19 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.000s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed 15:46:19 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000f90e20>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-rlzd6dwx88l5" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3097da51-addb-491f-9447-dc8f70dfa80a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3097da51-addb-491f-9447-dc8f70dfa80a", "name": "3097da51-addb-491f-9447-dc8f70dfa80a", "status": "Failed", "startTime": "2026-09-22T16:36:19.895011411Z", "endTime": "2026-09-22T17:05:24.149753544Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc000f90e20>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-rlzd6dwx88l5" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3097da51-addb-491f-9447-dc8f70dfa80a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3097da51-addb-491f-9447-dc8f70dfa80a", "name": "3097da51-addb-491f-9447-dc8f70dfa80a", "status": "Failed", "startTime": "2026-09-22T16:36:19.895011411Z", "endTime": "2026-09-22T17:05:24.149753544Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc00091bb80>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-q96f99txrjlq" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d26034f6-b329-4ae9-9bed-cb3c0606e891 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d26034f6-b329-4ae9-9bed-cb3c0606e891", "name": "d26034f6-b329-4ae9-9bed-cb3c0606e891", "status": "Failed", "startTime": "2026-09-22T16:37:47.299219553Z", "endTime": "2026-09-22T17:06:53.972888116Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc00091bb80>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-q96f99txrjlq" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d26034f6-b329-4ae9-9bed-cb3c0606e891 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d26034f6-b329-4ae9-9bed-cb3c0606e891", "name": "d26034f6-b329-4ae9-9bed-cb3c0606e891", "status": "Failed", "startTime": "2026-09-22T16:37:47.299219553Z", "endTime": "2026-09-22T17:06:53.972888116Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001082820>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-8sldg8lrtk8d" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c62e5766-be7f-4bc2-a830-e28cf7dd6767 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c62e5766-be7f-4bc2-a830-e28cf7dd6767", "name": "c62e5766-be7f-4bc2-a830-e28cf7dd6767", "status": "Failed", "startTime": "2026-09-22T16:35:50.929819703Z", "endTime": "2026-09-22T17:04:54.254355371Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001082820>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-8sldg8lrtk8d" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c62e5766-be7f-4bc2-a830-e28cf7dd6767 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c62e5766-be7f-4bc2-a830-e28cf7dd6767", "name": "c62e5766-be7f-4bc2-a830-e28cf7dd6767", "status": "Failed", "startTime": "2026-09-22T16:35:50.929819703Z", "endTime": "2026-09-22T17:04:54.254355371Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0xc0009066e0>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-x8qpd8lw49vm" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ee2f8ce5-9bed-48be-aeb0-cb61a7a4a93d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ee2f8ce5-9bed-48be-aeb0-cb61a7a4a93d", "name": "ee2f8ce5-9bed-48be-aeb0-cb61a7a4a93d", "status": "Failed", "startTime": "2026-09-22T16:36:12.016571667Z", "endTime": "2026-09-22T17:05:14.805027784Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0xc0009066e0>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-x8qpd8lw49vm" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ee2f8ce5-9bed-48be-aeb0-cb61a7a4a93d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ee2f8ce5-9bed-48be-aeb0-cb61a7a4a93d", "name": "ee2f8ce5-9bed-48be-aeb0-cb61a7a4a93d", "status": "Failed", "startTime": "2026-09-22T16:36:12.016571667Z", "endTime": "2026-09-22T17:05:14.805027784Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0017446a0>: failed to create HCP cluster np-dg-minor-8wc8tn: failed waiting for cluster="np-dg-minor-8wc8tn" in resourcegroup="rg-np-dg-minor-8wc8tn-22g8c5n8hhdx" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc38e9a7-2ed9-408e-a32b-6383c0505eaf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc38e9a7-2ed9-408e-a32b-6383c0505eaf", "name": "bc38e9a7-2ed9-408e-a32b-6383c0505eaf", "status": "Failed", "startTime": "2026-09-22T16:36:17.910968507Z", "endTime": "2026-09-22T17:05:24.133881522Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0017446a0>: failed to create HCP cluster np-dg-minor-8wc8tn: failed waiting for cluster="np-dg-minor-8wc8tn" in resourcegroup="rg-np-dg-minor-8wc8tn-22g8c5n8hhdx" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc38e9a7-2ed9-408e-a32b-6383c0505eaf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc38e9a7-2ed9-408e-a32b-6383c0505eaf", "name": "bc38e9a7-2ed9-408e-a32b-6383c0505eaf", "status": "Failed", "startTime": "2026-09-22T16:36:17.910968507Z", "endTime": "2026-09-22T17:05:24.133881522Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000cdea20>: failed to create HCP cluster np-downgrade-smt8b4: failed waiting for cluster="np-downgrade-smt8b4" in resourcegroup="rg-np-downgrade-smt8b4-p5xmqllchnwc" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7ee63103-3f10-4b67-a05a-795798c7c6c5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7ee63103-3f10-4b67-a05a-795798c7c6c5", "name": "7ee63103-3f10-4b67-a05a-795798c7c6c5", "status": "Failed", "startTime": "2026-09-22T16:36:26.373656246Z", "endTime": "2026-09-22T17:05:34.304731328Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000cdea20>: failed to create HCP cluster np-downgrade-smt8b4: failed waiting for cluster="np-downgrade-smt8b4" in resourcegroup="rg-np-downgrade-smt8b4-p5xmqllchnwc" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7ee63103-3f10-4b67-a05a-795798c7c6c5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7ee63103-3f10-4b67-a05a-795798c7c6c5", "name": "7ee63103-3f10-4b67-a05a-795798c7c6c5", "status": "Failed", "startTime": "2026-09-22T16:36:26.373656246Z", "endTime": "2026-09-22T17:05:34.304731328Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc000fdc6e0>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-zc4swxsfpjsc" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bfeaa5f3-bbb7-4a53-bbaf-a03f3b564a40 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bfeaa5f3-bbb7-4a53-bbaf-a03f3b564a40", "name": "bfeaa5f3-bbb7-4a53-bbaf-a03f3b564a40", "status": "Failed", "startTime": "2026-09-22T16:36:27.276605223Z", "endTime": "2026-09-22T17:05:34.330785345Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc000fdc6e0>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-zc4swxsfpjsc" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bfeaa5f3-bbb7-4a53-bbaf-a03f3b564a40 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bfeaa5f3-bbb7-4a53-bbaf-a03f3b564a40", "name": "bfeaa5f3-bbb7-4a53-bbaf-a03f3b564a40", "status": "Failed", "startTime": "2026-09-22T16:36:27.276605223Z", "endTime": "2026-09-22T17:05:34.330785345Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to cordon, drain, and uncordon a node in an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000d2ec20>: failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-kx88zsldj79h" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f0670065-55c5-420c-ad0f-2da7b18f6e3d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f0670065-55c5-420c-ad0f-2da7b18f6e3d", "name": "f0670065-55c5-420c-ad0f-2da7b18f6e3d", "status": "Failed", "startTime": "2026-09-22T16:36:26.610305452Z", "endTime": "2026-09-22T17:05:34.360693634Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000d2ec20>: failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-kx88zsldj79h" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f0670065-55c5-420c-ad0f-2da7b18f6e3d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f0670065-55c5-420c-ad0f-2da7b18f6e3d", "name": "f0670065-55c5-420c-ad0f-2da7b18f6e3d", "status": "Failed", "startTime": "2026-09-22T16:36:26.610305452Z", "endTime": "2026-09-22T17:05:34.360693634Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-jjgb97zqjz2j/cluster-candidate-4-21-gpl8gp should provision Unexpected error: <fmt.wrapError \| 0xc000ab88e0>: failed to create HCP cluster cluster-candidate-4-21-gpl8gp: failed waiting for cluster="cluster-candidate-4-21-gpl8gp" in resourcegroup="rg-candidate-4-21-jjgb97zqjz2j" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31cfd032-17f0-4e3f-8638-2dd7570fdc49 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31cfd032-17f0-4e3f-8638-2dd7570fdc49", "name": "31cfd032-17f0-4e3f-8638-2dd7570fdc49", "status": "Failed", "startTime": "2026-09-22T16:36:22.453860288Z", "endTime": "2026-09-22T17:05:24.148111981Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-jjgb97zqjz2j/cluster-candidate-4-21-gpl8gp should provision Unexpected error: <fmt.wrapError \| 0xc000ab88e0>: failed to create HCP cluster cluster-candidate-4-21-gpl8gp: failed waiting for cluster="cluster-candidate-4-21-gpl8gp" in resourcegroup="rg-candidate-4-21-jjgb97zqjz2j" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31cfd032-17f0-4e3f-8638-2dd7570fdc49 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31cfd032-17f0-4e3f-8638-2dd7570fdc49", "name": "31cfd032-17f0-4e3f-8638-2dd7570fdc49", "status": "Failed", "startTime": "2026-09-22T16:36:22.453860288Z", "endTime": "2026-09-22T17:05:24.148111981Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000aea5a0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-nptb84x7txrv" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9dca1d2-d589-496e-bb76-2944a2a36071 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9dca1d2-d589-496e-bb76-2944a2a36071", "name": "a9dca1d2-d589-496e-bb76-2944a2a36071", "status": "Failed", "startTime": "2026-09-22T16:37:41.11913075Z", "endTime": "2026-09-22T17:06:44.122367318Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000aea5a0>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-nptb84x7txrv" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9dca1d2-d589-496e-bb76-2944a2a36071 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a9dca1d2-d589-496e-bb76-2944a2a36071", "name": "a9dca1d2-d589-496e-bb76-2944a2a36071", "status": "Failed", "startTime": "2026-09-22T16:37:41.11913075Z", "endTime": "2026-09-22T17:06:44.122367318Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-gcds9cqvk6jx/cluster-candidate-4-20-lbcq4j should provision Unexpected error: <fmt.wrapError \| 0xc000fd81e0>: failed to create HCP cluster cluster-candidate-4-20-lbcq4j: failed waiting for cluster="cluster-candidate-4-20-lbcq4j" in resourcegroup="rg-candidate-4-20-gcds9cqvk6jx" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/58f294f8-155f-4a52-8225-6c82a85d1c93 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/58f294f8-155f-4a52-8225-6c82a85d1c93", "name": "58f294f8-155f-4a52-8225-6c82a85d1c93", "status": "Failed", "startTime": "2026-09-22T16:36:10.883840418Z", "endTime": "2026-09-22T17:05:14.699323611Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-gcds9cqvk6jx/cluster-candidate-4-20-lbcq4j should provision Unexpected error: <fmt.wrapError \| 0xc000fd81e0>: failed to create HCP cluster cluster-candidate-4-20-lbcq4j: failed waiting for cluster="cluster-candidate-4-20-lbcq4j" in resourcegroup="rg-candidate-4-20-gcds9cqvk6jx" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/58f294f8-155f-4a52-8225-6c82a85d1c93 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/58f294f8-155f-4a52-8225-6c82a85d1c93", "name": "58f294f8-155f-4a52-8225-6c82a85d1c93", "status": "Failed", "startTime": "2026-09-22T16:36:10.883840418Z", "endTime": "2026-09-22T17:05:14.699323611Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000abec20>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-85mvxbzslwt7" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b45304ce-67dd-49f1-b5d1-6815f1b21d81 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b45304ce-67dd-49f1-b5d1-6815f1b21d81", "name": "b45304ce-67dd-49f1-b5d1-6815f1b21d81", "status": "Failed", "startTime": "2026-09-22T16:36:20.392806615Z", "endTime": "2026-09-22T17:05:24.147500124Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000abec20>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-85mvxbzslwt7" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b45304ce-67dd-49f1-b5d1-6815f1b21d81 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b45304ce-67dd-49f1-b5d1-6815f1b21d81", "name": "b45304ce-67dd-49f1-b5d1-6815f1b21d81", "status": "Failed", "startTime": "2026-09-22T16:36:20.392806615Z", "endTime": "2026-09-22T17:05:24.147500124Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc001560120>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-j5gj726jttgj" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63296de6-e4f7-479c-8d94-e247b3fb7570 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63296de6-e4f7-479c-8d94-e247b3fb7570", "name": "63296de6-e4f7-479c-8d94-e247b3fb7570", "status": "Failed", "startTime": "2026-09-22T16:36:11.224909027Z", "endTime": "2026-09-22T17:05:14.704490055Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc001560120>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-j5gj726jttgj" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63296de6-e4f7-479c-8d94-e247b3fb7570 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63296de6-e4f7-479c-8d94-e247b3fb7570", "name": "63296de6-e4f7-479c-8d94-e247b3fb7570", "status": "Failed", "startTime": "2026-09-22T16:36:11.224909027Z", "endTime": "2026-09-22T17:05:14.704490055Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc0010f5240>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-89zmnzc9kp4x" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4644d955-2d8e-4777-9707-0e2d24d0ad08 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4644d955-2d8e-4777-9707-0e2d24d0ad08", "name": "4644d955-2d8e-4777-9707-0e2d24d0ad08", "status": "Failed", "startTime": "2026-09-22T16:36:11.010189501Z", "endTime": "2026-09-22T17:05:14.812991744Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc0010f5240>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-89zmnzc9kp4x" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4644d955-2d8e-4777-9707-0e2d24d0ad08 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4644d955-2d8e-4777-9707-0e2d24d0ad08", "name": "4644d955-2d8e-4777-9707-0e2d24d0ad08", "status": "Failed", "startTime": "2026-09-22T16:36:11.010189501Z", "endTime": "2026-09-22T17:05:14.812991744Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-wfcd7d78hw84/cluster-candidate-5-0-7q76df should provision Unexpected error: <fmt.wrapError \| 0xc000bbe7a0>: failed to create HCP cluster cluster-candidate-5-0-7q76df: failed waiting for cluster="cluster-candidate-5-0-7q76df" in resourcegroup="rg-candidate-5-0-wfcd7d78hw84" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6972e50c-f532-4d25-a01e-ed0f0fc64509 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6972e50c-f532-4d25-a01e-ed0f0fc64509", "name": "6972e50c-f532-4d25-a01e-ed0f0fc64509", "status": "Failed", "startTime": "2026-09-22T16:36:22.06647716Z", "endTime": "2026-09-22T17:05:24.024340059Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-wfcd7d78hw84/cluster-candidate-5-0-7q76df should provision Unexpected error: <fmt.wrapError \| 0xc000bbe7a0>: failed to create HCP cluster cluster-candidate-5-0-7q76df: failed waiting for cluster="cluster-candidate-5-0-7q76df" in resourcegroup="rg-candidate-5-0-wfcd7d78hw84" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6972e50c-f532-4d25-a01e-ed0f0fc64509 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6972e50c-f532-4d25-a01e-ed0f0fc64509", "name": "6972e50c-f532-4d25-a01e-ed0f0fc64509", "status": "Failed", "startTime": "2026-09-22T16:36:22.06647716Z", "endTime": "2026-09-22T17:05:24.024340059Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-qsl4cr" at install version 4.21 Unexpected error: <fmt.wrapError \| 0xc001056760>: failed to create HCP cluster cp-ystream-upgrade-4-22-qsl4cr: failed waiting for cluster="cp-ystream-upgrade-4-22-qsl4cr" in resourcegroup="rg-cp-ystream-upgrade-4-22-zvshks8mrjrp" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f41e8e09-6185-4b75-9296-bd49e5530f90 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f41e8e09-6185-4b75-9296-bd49e5530f90", "name": "f41e8e09-6185-4b75-9296-bd49e5530f90", "status": "Failed", "startTime": "2026-09-22T16:36:19.692646834Z", "endTime": "2026-09-22T17:05:24.125571007Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-qsl4cr" at install version 4.21 Unexpected error: <fmt.wrapError \| 0xc001056760>: failed to create HCP cluster cp-ystream-upgrade-4-22-qsl4cr: failed waiting for cluster="cp-ystream-upgrade-4-22-qsl4cr" in resourcegroup="rg-cp-ystream-upgrade-4-22-zvshks8mrjrp" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f41e8e09-6185-4b75-9296-bd49e5530f90 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f41e8e09-6185-4b75-9296-bd49e5530f90", "name": "f41e8e09-6185-4b75-9296-bd49e5530f90", "status": "Failed", "startTime": "2026-09-22T16:36:19.692646834Z", "endTime": "2026-09-22T17:05:24.125571007Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0xc000f24e20>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-m6tl2cdt492z" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1abffcc9-0969-4c8e-9169-58a7701f57c8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1abffcc9-0969-4c8e-9169-58a7701f57c8", "name": "1abffcc9-0969-4c8e-9169-58a7701f57c8", "status": "Failed", "startTime": "2026-09-22T16:37:10.047595495Z", "endTime": "2026-09-22T17:06:14.624098103Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0xc000f24e20>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-m6tl2cdt492z" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1abffcc9-0969-4c8e-9169-58a7701f57c8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1abffcc9-0969-4c8e-9169-58a7701f57c8", "name": "1abffcc9-0969-4c8e-9169-58a7701f57c8", "status": "Failed", "startTime": "2026-09-22T16:37:10.047595495Z", "endTime": "2026-09-22T17:06:14.624098103Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000e78f20>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-xndn9c5dzjz9" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e9538312-829d-4299-a8c1-299b80ee3f61 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e9538312-829d-4299-a8c1-299b80ee3f61", "name": "e9538312-829d-4299-a8c1-299b80ee3f61", "status": "Failed", "startTime": "2026-09-22T17:11:29.127000193Z", "endTime": "2026-09-22T17:40:29.489392482Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000e78f20>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-xndn9c5dzjz9" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e9538312-829d-4299-a8c1-299b80ee3f61 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e9538312-829d-4299-a8c1-299b80ee3f61", "name": "e9538312-829d-4299-a8c1-299b80ee3f61", "status": "Failed", "startTime": "2026-09-22T17:11:29.127000193Z", "endTime": "2026-09-22T17:40:29.489392482Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0xc001188420>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-2njgkd7pf75x" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/56b97f7e-7917-424a-8feb-3bbf833220a6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/56b97f7e-7917-424a-8feb-3bbf833220a6", "name": "56b97f7e-7917-424a-8feb-3bbf833220a6", "status": "Failed", "startTime": "2026-09-22T16:35:51.739370447Z", "endTime": "2026-09-22T17:04:54.27229918Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0xc001188420>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-2njgkd7pf75x" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/56b97f7e-7917-424a-8feb-3bbf833220a6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/56b97f7e-7917-424a-8feb-3bbf833220a6", "name": "56b97f7e-7917-424a-8feb-3bbf833220a6", "status": "Failed", "startTime": "2026-09-22T16:35:51.739370447Z", "endTime": "2026-09-22T17:04:54.27229918Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc0008c05e0>: failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-q22cpt6sfmdd" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42a9d035-c8f7-4515-96cd-4a75bad14685 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42a9d035-c8f7-4515-96cd-4a75bad14685", "name": "42a9d035-c8f7-4515-96cd-4a75bad14685", "status": "Failed", "startTime": "2026-09-22T16:36:26.805037234Z", "endTime": "2026-09-22T17:05:34.371586074Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc0008c05e0>: failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-q22cpt6sfmdd" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42a9d035-c8f7-4515-96cd-4a75bad14685 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42a9d035-c8f7-4515-96cd-4a75bad14685", "name": "42a9d035-c8f7-4515-96cd-4a75bad14685", "status": "Failed", "startTime": "2026-09-22T16:36:26.805037234Z", "endTime": "2026-09-22T17:05:34.371586074Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-46r9kp" with version 4.21.33 on candidate channel Unexpected error: <fmt.wrapError \| 0xc001a72500>: failed to create HCP cluster cluster-zstream-4-21-46r9kp: failed waiting for cluster="cluster-zstream-4-21-46r9kp" in resourcegroup="rg-zstream-upgrade-4-21-kqtbx7l8f2cd" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c40ffce-91f8-432f-98a9-a17f456386e1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c40ffce-91f8-432f-98a9-a17f456386e1", "name": "7c40ffce-91f8-432f-98a9-a17f456386e1", "status": "Failed", "startTime": "2026-09-22T16:36:11.107965605Z", "endTime": "2026-09-22T17:05:14.587756636Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-46r9kp" with version 4.21.33 on candidate channel Unexpected error: <fmt.wrapError \| 0xc001a72500>: failed to create HCP cluster cluster-zstream-4-21-46r9kp: failed waiting for cluster="cluster-zstream-4-21-46r9kp" in resourcegroup="rg-zstream-upgrade-4-21-kqtbx7l8f2cd" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c40ffce-91f8-432f-98a9-a17f456386e1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7c40ffce-91f8-432f-98a9-a17f456386e1", "name": "7c40ffce-91f8-432f-98a9-a17f456386e1", "status": "Failed", "startTime": "2026-09-22T16:36:11.107965605Z", "endTime": "2026-09-22T17:05:14.587756636Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-7mgdpk at version 4.22.14 Unexpected error: <fmt.wrapError \| 0xc000e0e240>: failed to create HCP cluster np-skip-minor-7mgdpk: failed waiting for cluster="np-skip-minor-7mgdpk" in resourcegroup="rg-np-skip-minor-7mgdpk-zppwbmg2pxz9" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d63e95f6-06b8-4ecd-adcb-ca227f3cf447 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d63e95f6-06b8-4ecd-adcb-ca227f3cf447", "name": "d63e95f6-06b8-4ecd-adcb-ca227f3cf447", "status": "Failed", "startTime": "2026-09-22T16:36:28.13810987Z", "endTime": "2026-09-22T17:05:34.35992663Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-7mgdpk at version 4.22.14 Unexpected error: <fmt.wrapError \| 0xc000e0e240>: failed to create HCP cluster np-skip-minor-7mgdpk: failed waiting for cluster="np-skip-minor-7mgdpk" in resourcegroup="rg-np-skip-minor-7mgdpk-zppwbmg2pxz9" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d63e95f6-06b8-4ecd-adcb-ca227f3cf447 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d63e95f6-06b8-4ecd-adcb-ca227f3cf447", "name": "d63e95f6-06b8-4ecd-adcb-ca227f3cf447", "status": "Failed", "startTime": "2026-09-22T16:36:28.13810987Z", "endTime": "2026-09-22T17:05:34.35992663Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000afc5e0>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-km6krsmrn4pb" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f22da8f-3a4b-46a6-9028-2313648c12c9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f22da8f-3a4b-46a6-9028-2313648c12c9", "name": "1f22da8f-3a4b-46a6-9028-2313648c12c9", "status": "Failed", "startTime": "2026-09-22T16:36:15.314214518Z", "endTime": "2026-09-22T17:05:24.094562064Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000afc5e0>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-km6krsmrn4pb" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f22da8f-3a4b-46a6-9028-2313648c12c9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f22da8f-3a4b-46a6-9028-2313648c12c9", "name": "1f22da8f-3a4b-46a6-9028-2313648c12c9", "status": "Failed", "startTime": "2026-09-22T16:36:15.314214518Z", "endTime": "2026-09-22T17:05:24.094562064Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--q8pfz7dcxcwn/cluster-hs-pre-6s5mlg should provision Unexpected error: <fmt.wrapError \| 0xc00110ee80>: failed to create HCP cluster cluster-hs-pre-6s5mlg: failed waiting for cluster="cluster-hs-pre-6s5mlg" in resourcegroup="rg-hs-pre--q8pfz7dcxcwn" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/953251c4-2f9e-4fc4-8a41-b0e88bd1dfe6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/953251c4-2f9e-4fc4-8a41-b0e88bd1dfe6", "name": "953251c4-2f9e-4fc4-8a41-b0e88bd1dfe6", "status": "Failed", "startTime": "2026-09-22T16:36:11.476929134Z", "endTime": "2026-09-22T17:05:14.534373591Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--q8pfz7dcxcwn/cluster-hs-pre-6s5mlg should provision Unexpected error: <fmt.wrapError \| 0xc00110ee80>: failed to create HCP cluster cluster-hs-pre-6s5mlg: failed waiting for cluster="cluster-hs-pre-6s5mlg" in resourcegroup="rg-hs-pre--q8pfz7dcxcwn" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/953251c4-2f9e-4fc4-8a41-b0e88bd1dfe6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/953251c4-2f9e-4fc4-8a41-b0e88bd1dfe6", "name": "953251c4-2f9e-4fc4-8a41-b0e88bd1dfe6", "status": "Failed", "startTime": "2026-09-22T16:36:11.476929134Z", "endTime": "2026-09-22T17:05:14.534373591Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-gb92sf with version 4.20.39 Unexpected error: <fmt.wrapError \| 0xc000ac37a0>: failed to create HCP cluster np-version-upgrade-cluster-gb92sf: failed waiting for cluster="np-version-upgrade-cluster-gb92sf" in resourcegroup="rg-np-version-upgrade-gb92sf-xwrmnplklbpq" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/99bfd1c8-1de8-4bb4-9f76-bdbfa324ac04 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/99bfd1c8-1de8-4bb4-9f76-bdbfa324ac04", "name": "99bfd1c8-1de8-4bb4-9f76-bdbfa324ac04", "status": "Failed", "startTime": "2026-09-22T16:36:34.388273814Z", "endTime": "2026-09-22T17:05:34.376441059Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-gb92sf with version 4.20.39 Unexpected error: <fmt.wrapError \| 0xc000ac37a0>: failed to create HCP cluster np-version-upgrade-cluster-gb92sf: failed waiting for cluster="np-version-upgrade-cluster-gb92sf" in resourcegroup="rg-np-version-upgrade-gb92sf-xwrmnplklbpq" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/99bfd1c8-1de8-4bb4-9f76-bdbfa324ac04 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/99bfd1c8-1de8-4bb4-9f76-bdbfa324ac04", "name": "99bfd1c8-1de8-4bb4-9f76-bdbfa324ac04", "status": "Failed", "startTime": "2026-09-22T16:36:34.388273814Z", "endTime": "2026-09-22T17:05:34.376441059Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc000d9f1a0>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-kgwvsmk5rm2h" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a6993d29-caec-456b-8254-690fe28ba79d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a6993d29-caec-456b-8254-690fe28ba79d", "name": "a6993d29-caec-456b-8254-690fe28ba79d", "status": "Failed", "startTime": "2026-09-22T16:36:27.150078447Z", "endTime": "2026-09-22T17:05:34.293937324Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc000d9f1a0>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-kgwvsmk5rm2h" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a6993d29-caec-456b-8254-690fe28ba79d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a6993d29-caec-456b-8254-690fe28ba79d", "name": "a6993d29-caec-456b-8254-690fe28ba79d", "status": "Failed", "startTime": "2026-09-22T16:36:27.150078447Z", "endTime": "2026-09-22T17:05:34.293937324Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc001086f20>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-s4xt9ndfdfqn" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95ac10fd-a19e-4b3d-a436-7176e2fd4f08 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95ac10fd-a19e-4b3d-a436-7176e2fd4f08", "name": "95ac10fd-a19e-4b3d-a436-7176e2fd4f08", "status": "Failed", "startTime": "2026-09-22T16:38:42.665905872Z", "endTime": "2026-09-22T17:07:49.360946035Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc001086f20>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-s4xt9ndfdfqn" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95ac10fd-a19e-4b3d-a436-7176e2fd4f08 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95ac10fd-a19e-4b3d-a436-7176e2fd4f08", "name": "95ac10fd-a19e-4b3d-a436-7176e2fd4f08", "status": "Failed", "startTime": "2026-09-22T16:38:42.665905872Z", "endTime": "2026-09-22T17:07:49.360946035Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-mmf24lvbl69k/cluster-candidate-4-22-jq772f should provision Unexpected error: <fmt.wrapError \| 0xc001116360>: failed to create HCP cluster cluster-candidate-4-22-jq772f: failed waiting for cluster="cluster-candidate-4-22-jq772f" in resourcegroup="rg-candidate-4-22-mmf24lvbl69k" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d37ba33c-41ef-4bc9-b99d-39c30ef70d7f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d37ba33c-41ef-4bc9-b99d-39c30ef70d7f", "name": "d37ba33c-41ef-4bc9-b99d-39c30ef70d7f", "status": "Failed", "startTime": "2026-09-22T16:36:11.574580339Z", "endTime": "2026-09-22T17:05:15.304759832Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-mmf24lvbl69k/cluster-candidate-4-22-jq772f should provision Unexpected error: <fmt.wrapError \| 0xc001116360>: failed to create HCP cluster cluster-candidate-4-22-jq772f: failed waiting for cluster="cluster-candidate-4-22-jq772f" in resourcegroup="rg-candidate-4-22-mmf24lvbl69k" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d37ba33c-41ef-4bc9-b99d-39c30ef70d7f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d37ba33c-41ef-4bc9-b99d-39c30ef70d7f", "name": "d37ba33c-41ef-4bc9-b99d-39c30ef70d7f", "status": "Failed", "startTime": "2026-09-22T16:36:11.574580339Z", "endTime": "2026-09-22T17:05:15.304759832Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:88]: failed to create HCP cluster for kusto logs test Unexpected error: <fmt.wrapError \| 0xc0014b0700>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-cth475kbhvnz" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ad99362b-03f1-478d-8c68-903b1a5b453e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ad99362b-03f1-478d-8c68-903b1a5b453e", "name": "ad99362b-03f1-478d-8c68-903b1a5b453e", "status": "Failed", "startTime": "2026-09-22T16:36:15.474739273Z", "endTime": "2026-09-22T17:05:24.196969347Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:88]: failed to create HCP cluster for kusto logs test Unexpected error: <fmt.wrapError \| 0xc0014b0700>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-cth475kbhvnz" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ad99362b-03f1-478d-8c68-903b1a5b453e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ad99362b-03f1-478d-8c68-903b1a5b453e", "name": "ad99362b-03f1-478d-8c68-903b1a5b453e", "status": "Failed", "startTime": "2026-09-22T16:36:15.474739273Z", "endTime": "2026-09-22T17:05:24.196969347Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:83]: failed to create HCP cluster np-update-nodes-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000c521c0>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-8ql5kd5f7tjm" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/547703e0-0d1c-43cc-a185-feb70bc56c33 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/547703e0-0d1c-43cc-a185-feb70bc56c33", "name": "547703e0-0d1c-43cc-a185-feb70bc56c33", "status": "Failed", "startTime": "2026-09-22T16:36:27.4357733Z", "endTime": "2026-09-22T17:05:34.221694646Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:83]: failed to create HCP cluster np-update-nodes-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000c521c0>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-8ql5kd5f7tjm" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/547703e0-0d1c-43cc-a185-feb70bc56c33 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/547703e0-0d1c-43cc-a185-feb70bc56c33", "name": "547703e0-0d1c-43cc-a185-feb70bc56c33", "status": "Failed", "startTime": "2026-09-22T16:36:27.4357733Z", "endTime": "2026-09-22T17:05:34.221694646Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc000fc8a80>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-rv2rq259ntc8" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/94923f6a-6919-48c0-80e9-3c3b9c3730d1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/94923f6a-6919-48c0-80e9-3c3b9c3730d1", "name": "94923f6a-6919-48c0-80e9-3c3b9c3730d1", "status": "Failed", "startTime": "2026-09-22T16:35:56.097535459Z", "endTime": "2026-09-22T17:05:04.00508538Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc000fc8a80>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-rv2rq259ntc8" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/94923f6a-6919-48c0-80e9-3c3b9c3730d1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/94923f6a-6919-48c0-80e9-3c3b9c3730d1", "name": "94923f6a-6919-48c0-80e9-3c3b9c3730d1", "status": "Failed", "startTime": "2026-09-22T16:35:56.097535459Z", "endTime": "2026-09-22T17:05:04.00508538Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-r2ls78" with version 4.22.13 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000b566a0>: failed to create HCP cluster cluster-zstream-4-22-r2ls78: failed waiting for cluster="cluster-zstream-4-22-r2ls78" in resourcegroup="rg-zstream-upgrade-4-22-sm79xt4bwmhx" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/55386486-26be-4228-bb22-1d85147aacf3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/55386486-26be-4228-bb22-1d85147aacf3", "name": "55386486-26be-4228-bb22-1d85147aacf3", "status": "Failed", "startTime": "2026-09-22T16:36:16.148910931Z", "endTime": "2026-09-22T17:05:24.221837381Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-r2ls78" with version 4.22.13 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000b566a0>: failed to create HCP cluster cluster-zstream-4-22-r2ls78: failed waiting for cluster="cluster-zstream-4-22-r2ls78" in resourcegroup="rg-zstream-upgrade-4-22-sm79xt4bwmhx" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/55386486-26be-4228-bb22-1d85147aacf3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/55386486-26be-4228-bb22-1d85147aacf3", "name": "55386486-26be-4228-bb22-1d85147aacf3", "status": "Failed", "startTime": "2026-09-22T16:36:16.148910931Z", "endTime": "2026-09-22T17:05:24.221837381Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-b7bkwk" with version 4.20.38 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00093c5c0>: failed to create HCP cluster cluster-zstream-4-20-b7bkwk: failed waiting for cluster="cluster-zstream-4-20-b7bkwk" in resourcegroup="rg-zstream-upgrade-4-20-j6h5fx5bsw4s" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01b04d32-39e2-487e-ae84-b4c3e4359648 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01b04d32-39e2-487e-ae84-b4c3e4359648", "name": "01b04d32-39e2-487e-ae84-b4c3e4359648", "status": "Failed", "startTime": "2026-09-22T16:36:21.363686989Z", "endTime": "2026-09-22T17:05:24.227617931Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-b7bkwk" with version 4.20.38 on candidate channel Unexpected error: <fmt.wrapError \| 0xc00093c5c0>: failed to create HCP cluster cluster-zstream-4-20-b7bkwk: failed waiting for cluster="cluster-zstream-4-20-b7bkwk" in resourcegroup="rg-zstream-upgrade-4-20-j6h5fx5bsw4s" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01b04d32-39e2-487e-ae84-b4c3e4359648 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/01b04d32-39e2-487e-ae84-b4c3e4359648", "name": "01b04d32-39e2-487e-ae84-b4c3e4359648", "status": "Failed", "startTime": "2026-09-22T16:36:21.363686989Z", "endTime": "2026-09-22T17:05:24.227617931Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc000a425a0>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-knnjcpzbn8r2" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb18f2ee-8f38-48b6-9d86-84b032732a97 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb18f2ee-8f38-48b6-9d86-84b032732a97", "name": "cb18f2ee-8f38-48b6-9d86-84b032732a97", "status": "Failed", "startTime": "2026-09-22T16:36:37.699532367Z", "endTime": "2026-09-22T17:05:44.628988758Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc000a425a0>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-knnjcpzbn8r2" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb18f2ee-8f38-48b6-9d86-84b032732a97 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb18f2ee-8f38-48b6-9d86-84b032732a97", "name": "cb18f2ee-8f38-48b6-9d86-84b032732a97", "status": "Failed", "startTime": "2026-09-22T16:36:37.699532367Z", "endTime": "2026-09-22T17:05:44.628988758Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000ae0ce0>: failed to create HCP cluster ea-listsfgcb9: failed waiting for cluster="ea-listsfgcb9" in resourcegroup="ea-list-m8pl2s5stks2" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1e9b63b5-90b2-40fb-859e-8c4e24d4ec59 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1e9b63b5-90b2-40fb-859e-8c4e24d4ec59", "name": "1e9b63b5-90b2-40fb-859e-8c4e24d4ec59", "status": "Failed", "startTime": "2026-09-22T16:36:20.544914102Z", "endTime": "2026-09-22T17:05:24.196772116Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000ae0ce0>: failed to create HCP cluster ea-listsfgcb9: failed waiting for cluster="ea-listsfgcb9" in resourcegroup="ea-list-m8pl2s5stks2" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1e9b63b5-90b2-40fb-859e-8c4e24d4ec59 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1e9b63b5-90b2-40fb-859e-8c4e24d4ec59", "name": "1e9b63b5-90b2-40fb-859e-8c4e24d4ec59", "status": "Failed", "startTime": "2026-09-22T16:36:20.544914102Z", "endTime": "2026-09-22T17:05:24.196772116Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc0008fe6a0>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-xxj2hlgdfm4c" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2b9400a7-5197-4acd-b7f8-f4eb830bfc0b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2b9400a7-5197-4acd-b7f8-f4eb830bfc0b", "name": "2b9400a7-5197-4acd-b7f8-f4eb830bfc0b", "status": "Failed", "startTime": "2026-09-22T16:36:06.037503121Z", "endTime": "2026-09-22T17:05:15.290687493Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc0008fe6a0>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-xxj2hlgdfm4c" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2b9400a7-5197-4acd-b7f8-f4eb830bfc0b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2b9400a7-5197-4acd-b7f8-f4eb830bfc0b", "name": "2b9400a7-5197-4acd-b7f8-f4eb830bfc0b", "status": "Failed", "startTime": "2026-09-22T16:36:06.037503121Z", "endTime": "2026-09-22T17:05:15.290687493Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc00015a360>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-9q4d8jcrrwfp" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a2c8014a-b687-41fd-ace7-be97be13a6e4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a2c8014a-b687-41fd-ace7-be97be13a6e4", "name": "a2c8014a-b687-41fd-ace7-be97be13a6e4", "status": "Failed", "startTime": "2026-09-22T16:36:10.297297909Z", "endTime": "2026-09-22T17:05:14.590541785Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc00015a360>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-9q4d8jcrrwfp" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a2c8014a-b687-41fd-ace7-be97be13a6e4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a2c8014a-b687-41fd-ace7-be97be13a6e4", "name": "a2c8014a-b687-41fd-ace7-be97be13a6e4", "status": "Failed", "startTime": "2026-09-22T16:36:10.297297909Z", "endTime": "2026-09-22T17:05:14.590541785Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-z2p8kx with version 4.21.34 Unexpected error: <fmt.wrapError \| 0xc0017febe0>: failed to create HCP cluster np-version-upgrade-cluster-z2p8kx: failed waiting for cluster="np-version-upgrade-cluster-z2p8kx" in resourcegroup="rg-np-version-upgrade-z2p8kx-6psgzckkqp6m" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc9376d9-c75f-49c8-b375-fda4aabeb8cf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc9376d9-c75f-49c8-b375-fda4aabeb8cf", "name": "bc9376d9-c75f-49c8-b375-fda4aabeb8cf", "status": "Failed", "startTime": "2026-09-22T16:36:27.960631712Z", "endTime": "2026-09-22T17:05:34.274798638Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-z2p8kx with version 4.21.34 Unexpected error: <fmt.wrapError \| 0xc0017febe0>: failed to create HCP cluster np-version-upgrade-cluster-z2p8kx: failed waiting for cluster="np-version-upgrade-cluster-z2p8kx" in resourcegroup="rg-np-version-upgrade-z2p8kx-6psgzckkqp6m" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc9376d9-c75f-49c8-b375-fda4aabeb8cf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bc9376d9-c75f-49c8-b375-fda4aabeb8cf", "name": "bc9376d9-c75f-49c8-b375-fda4aabeb8cf", "status": "Failed", "startTime": "2026-09-22T16:36:27.960631712Z", "endTime": "2026-09-22T17:05:34.274798638Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 15:46:19 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20 Unexpected error: <fmt.wrapErrors \| 0xc000af31d0>: failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-r6xxwmvr4vzb" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-r6xxwmvr4vzb, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20 Unexpected error: <fmt.wrapErrors \| 0xc000af31d0>: failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-r6xxwmvr4vzb" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-r6xxwmvr4vzb, error: context deadline exceeded ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000e5a440>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-dhfmcr22g2ft" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a322f570-1cad-47c7-8044-b71331609113 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a322f570-1cad-47c7-8044-b71331609113", "name": "a322f570-1cad-47c7-8044-b71331609113", "status": "Failed", "startTime": "2026-09-22T16:36:10.925233852Z", "endTime": "2026-09-22T17:05:15.583156498Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000e5a440>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-dhfmcr22g2ft" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a322f570-1cad-47c7-8044-b71331609113 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a322f570-1cad-47c7-8044-b71331609113", "name": "a322f570-1cad-47c7-8044-b71331609113", "status": "Failed", "startTime": "2026-09-22T16:36:10.925233852Z", "endTime": "2026-09-22T17:05:15.583156498Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-5g8jcf" at install version 4.20 Unexpected error: <fmt.wrapError \| 0xc000e32c60>: failed to create HCP cluster cp-ystream-upgrade-4-21-5g8jcf: failed waiting for cluster="cp-ystream-upgrade-4-21-5g8jcf" in resourcegroup="rg-cp-ystream-upgrade-4-21-6hsjb89rx8zt" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31b29abc-8d4e-4d29-ac99-57c16217a5ef -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31b29abc-8d4e-4d29-ac99-57c16217a5ef", "name": "31b29abc-8d4e-4d29-ac99-57c16217a5ef", "status": "Failed", "startTime": "2026-09-22T16:36:18.914832024Z", "endTime": "2026-09-22T17:05:24.213170431Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-5g8jcf" at install version 4.20 Unexpected error: <fmt.wrapError \| 0xc000e32c60>: failed to create HCP cluster cp-ystream-upgrade-4-21-5g8jcf: failed waiting for cluster="cp-ystream-upgrade-4-21-5g8jcf" in resourcegroup="rg-cp-ystream-upgrade-4-21-6hsjb89rx8zt" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31b29abc-8d4e-4d29-ac99-57c16217a5ef -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/31b29abc-8d4e-4d29-ac99-57c16217a5ef", "name": "31b29abc-8d4e-4d29-ac99-57c16217a5ef", "status": "Failed", "startTime": "2026-09-22T16:36:18.914832024Z", "endTime": "2026-09-22T17:05:24.213170431Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-lktj2c" with version 5.0.0-rc.2 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0010e2d40>: failed to create HCP cluster cluster-zstream-5-0-lktj2c: failed waiting for cluster="cluster-zstream-5-0-lktj2c" in resourcegroup="rg-zstream-upgrade-5-0-nwzl7cn6kqp6" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8b19ecd1-cdce-44ce-9718-67cf7c27e00a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8b19ecd1-cdce-44ce-9718-67cf7c27e00a", "name": "8b19ecd1-cdce-44ce-9718-67cf7c27e00a", "status": "Failed", "startTime": "2026-09-22T16:36:22.485113932Z", "endTime": "2026-09-22T17:05:24.233153854Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-lktj2c" with version 5.0.0-rc.2 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0010e2d40>: failed to create HCP cluster cluster-zstream-5-0-lktj2c: failed waiting for cluster="cluster-zstream-5-0-lktj2c" in resourcegroup="rg-zstream-upgrade-5-0-nwzl7cn6kqp6" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8b19ecd1-cdce-44ce-9718-67cf7c27e00a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8b19ecd1-cdce-44ce-9718-67cf7c27e00a", "name": "8b19ecd1-cdce-44ce-9718-67cf7c27e00a", "status": "Failed", "startTime": "2026-09-22T16:36:22.485113932Z", "endTime": "2026-09-22T17:05:24.233153854Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-66g59p" at install version 4.22 Unexpected error: <fmt.wrapError \| 0xc000d3ea40>: failed to create HCP cluster cp-ystream-upgrade-5-0-66g59p: failed waiting for cluster="cp-ystream-upgrade-5-0-66g59p" in resourcegroup="rg-cp-ystream-upgrade-5-0-dmnztktbvcnh" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/89a5b6f9-6cb3-484b-b3ee-cf482e31ecb8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/89a5b6f9-6cb3-484b-b3ee-cf482e31ecb8", "name": "89a5b6f9-6cb3-484b-b3ee-cf482e31ecb8", "status": "Failed", "startTime": "2026-09-22T16:36:18.511400811Z", "endTime": "2026-09-22T17:05:24.116275999Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-66g59p" at install version 4.22 Unexpected error: <fmt.wrapError \| 0xc000d3ea40>: failed to create HCP cluster cp-ystream-upgrade-5-0-66g59p: failed waiting for cluster="cp-ystream-upgrade-5-0-66g59p" in resourcegroup="rg-cp-ystream-upgrade-5-0-dmnztktbvcnh" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/89a5b6f9-6cb3-484b-b3ee-cf482e31ecb8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/89a5b6f9-6cb3-484b-b3ee-cf482e31ecb8", "name": "89a5b6f9-6cb3-484b-b3ee-cf482e31ecb8", "status": "Failed", "startTime": "2026-09-22T16:36:18.511400811Z", "endTime": "2026-09-22T17:05:24.116275999Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000f670a0>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-kcthj55945nv" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/591c7311-7de3-4e2e-b8bb-fbe0b4624e66 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/591c7311-7de3-4e2e-b8bb-fbe0b4624e66", "name": "591c7311-7de3-4e2e-b8bb-fbe0b4624e66", "status": "Failed", "startTime": "2026-09-22T16:37:05.62592778Z", "endTime": "2026-09-22T17:06:14.174276347Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000f670a0>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-kcthj55945nv" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/591c7311-7de3-4e2e-b8bb-fbe0b4624e66 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/591c7311-7de3-4e2e-b8bb-fbe0b4624e66", "name": "591c7311-7de3-4e2e-b8bb-fbe0b4624e66", "status": "Failed", "startTime": "2026-09-22T16:37:05.62592778Z", "endTime": "2026-09-22T17:06:14.174276347Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000ed4140>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-7f9fwfzj96zl" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/48240b99-a128-4809-98a9-f9a52f0f247e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/48240b99-a128-4809-98a9-f9a52f0f247e", "name": "48240b99-a128-4809-98a9-f9a52f0f247e", "status": "Failed", "startTime": "2026-09-22T16:36:22.143851357Z", "endTime": "2026-09-22T17:05:24.113006702Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000ed4140>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-7f9fwfzj96zl" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/48240b99-a128-4809-98a9-f9a52f0f247e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/48240b99-a128-4809-98a9-f9a52f0f247e", "name": "48240b99-a128-4809-98a9-f9a52f0f247e", "status": "Failed", "startTime": "2026-09-22T16:36:22.143851357Z", "endTime": "2026-09-22T17:05:24.113006702Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 15:46:19 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000c14a40>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-62972sjbd4tf" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb15bb25-f3ed-4ac7-a122-ae29b89ccd00 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb15bb25-f3ed-4ac7-a122-ae29b89ccd00", "name": "eb15bb25-f3ed-4ac7-a122-ae29b89ccd00", "status": "Failed", "startTime": "2026-09-22T16:36:20.790218421Z", "endTime": "2026-09-22T17:05:24.226195914Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000c14a40>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-62972sjbd4tf" to finish creating: GET https://rp.j4475392.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb15bb25-f3ed-4ac7-a122-ae29b89ccd00 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/eb15bb25-f3ed-4ac7-a122-ae29b89ccd00", "name": "eb15bb25-f3ed-4ac7-a122-ae29b89ccd00", "status": "Failed", "startTime": "2026-09-22T16:36:20.790218421Z", "endTime": "2026-09-22T17:05:24.226195914Z", "error": { "code": "InternalServerError", "message": "cluster creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred alert (4) alert [hcp] KubeDeploymentReplicasMismatch fired 15:46:19 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 4 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T17:43:54Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4475392-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.3:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/hcps-j4475392", namespace="ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m", pod="arohcp-monitor-kube-state-metrics-7955558448-pwzhh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m/cluster-api has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Fired (not resolved) Started: 2026-09-22T17:43:54Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4475392-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.3:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/hcps-j4475392", namespace="ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m", pod="arohcp-monitor-kube-state-metrics-7955558448-pwzhh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 3: State: Fired (not resolved) Started: 2026-09-22T17:45:55Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4475392-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.3:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/hcps-j4475392", namespace="ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m", pod="arohcp-monitor-kube-state-metrics-7955558448-pwzhh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m/cluster-api has not matched the expected number of replicas for longer than 30 minutes. Firing 4: State: Fired (not resolved) Started: 2026-09-22T17:45:55Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j4475392-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.3:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/hcps-j4475392", namespace="ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m", pod="arohcp-monitor-kube-state-metrics-7955558448-pwzhh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2t06119diqdpvr443dmiciulp17qu6qb-b8x0f3j9h1k5s1m/capi-provider has not matched the expected number of replicas for longer than 30 minutes. alert [svc] userJourneyClusterProvisionErrors6h30m fired 15:46:19 UTC · failed at alert · test [aro-hcp-observability] [svc] alert userJourneyClusterProvisionErrors6h30m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T17:38:22Z Severity: Sev3 Labels: alertname="userJourneyClusterProvisionErrors6h30m", cluster="ci01-j4475392-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/services-j4475392", region="westus3", severity="3", short_window="30m", slo="cluster-provision-errors" Description: More than 30% of completed cluster create (install) operations on ci01-j4475392-svc failed over the last 6 hours (at least 5 completions), a 6x burn of the 95% SLO budget. alert [svc] ServiceMemoryDrift fired 15:46:19 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T17:49:51Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j4475392-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-ff8gg", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-ff8gg (namespace arobit) on cluster ci01-j4475392-svc has averaged 123.8% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml. alert [svc] userJourneyClusterProvisionErrors1h5m fired 15:46:19 UTC · failed at alert · test [aro-hcp-observability] [svc] alert userJourneyClusterProvisionErrors1h5m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T17:14:23Z Severity: Sev3 Labels: alertname="userJourneyClusterProvisionErrors1h5m", cluster="ci01-j4475392-svc", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4475392/providers/microsoft.monitor/accounts/services-j4475392", region="westus3", severity="3", short_window="5m", slo="cluster-provision-errors" Description: More than 72% of completed cluster create (install) operations on ci01-j4475392-svc failed over the last hour with at least 3 failures, a 14.4x burn of the 95% SLO budget.
15:37:03	2h 13m 43s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head e8df860 · merge e8df860	canadacentral	alert, e2e	failed	#5846 (merged)	4	e2e (3) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:37:03 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a3e0f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-sxrknhrcf2sv" to finish deleting: GET https://rp.j0959488.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d6a496d1-7d6a-46b3-b956-b999573cbcee -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d6a496d1-7d6a-46b3-b956-b999573cbcee", "name": "d6a496d1-7d6a-46b3-b956-b999573cbcee", "status": "Failed", "startTime": "2026-09-22T17:10:29.991328675Z", "endTime": "2026-09-22T17:34:37.126574936Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t05q325vna98avgbb2eeekk2eu34dgn still exists (deletion dispatched at 2026-09-22T17:10:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a3e0f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-sxrknhrcf2sv" to finish deleting: GET https://rp.j0959488.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d6a496d1-7d6a-46b3-b956-b999573cbcee -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d6a496d1-7d6a-46b3-b956-b999573cbcee", "name": "d6a496d1-7d6a-46b3-b956-b999573cbcee", "status": "Failed", "startTime": "2026-09-22T17:10:29.991328675Z", "endTime": "2026-09-22T17:34:37.126574936Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t05q325vna98avgbb2eeekk2eu34dgn still exists (deletion dispatched at 2026-09-22T17:10:31Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:37:03 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00177c1c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-wglcgf" in resourcegroup="rg-np-version-upgrade-wglcgf-xqkdchgrl5rp" to finish deleting: GET https://rp.j0959488.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/88d50275-408d-4ac0-97bc-7497d8fe6fd4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/88d50275-408d-4ac0-97bc-7497d8fe6fd4", "name": "88d50275-408d-4ac0-97bc-7497d8fe6fd4", "status": "Failed", "startTime": "2026-09-22T17:00:36.387615721Z", "endTime": "2026-09-22T17:24:37.112980505Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t05q51hjkkjt4r1accultpettv5d9l7 still exists (deletion dispatched at 2026-09-22T17:00:38Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00177c1c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-wglcgf" in resourcegroup="rg-np-version-upgrade-wglcgf-xqkdchgrl5rp" to finish deleting: GET https://rp.j0959488.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/88d50275-408d-4ac0-97bc-7497d8fe6fd4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/88d50275-408d-4ac0-97bc-7497d8fe6fd4", "name": "88d50275-408d-4ac0-97bc-7497d8fe6fd4", "status": "Failed", "startTime": "2026-09-22T17:00:36.387615721Z", "endTime": "2026-09-22T17:24:37.112980505Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t05q51hjkkjt4r1accultpettv5d9l7 still exists (deletion dispatched at 2026-09-22T17:00:38Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist 15:37:03 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.002s. Certificate should be issued by Let's Encrypt staging via ACME DNS-01 Expected success, but got an error: <errors.errorString \| 0xc0018dd510>: certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.002s. Certificate should be issued by Let's Encrypt staging via ACME DNS-01 Expected success, but got an error: <errors.errorString \| 0xc0018dd510>: certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist ... alert (1) alert [svc] ServiceMemoryDrift fired 15:37:03 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T17:28:44Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j0959488-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-kwb9z", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-kwb9z (namespace arobit) on cluster ci01-j0959488-svc has averaged 127.5% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
15:31:45	12m 54s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head aa66f79	n/a	unknown	failed	#5493 (open)	1	non-artifact-backed failure (no junit artifacts)
15:00:34	2h 16m 46s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 54049e5 · merge 54049e5	westus3	alert, e2e	failed	#6995 (merged)	5	e2e (4) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:00:34 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0010f82a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-5s7lbj2gl95t" to finish deleting: GET https://rp.j2605440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f29ddf1-5f30-4805-8772-bb86c797c186 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f29ddf1-5f30-4805-8772-bb86c797c186", "name": "1f29ddf1-5f30-4805-8772-bb86c797c186", "status": "Failed", "startTime": "2026-09-22T16:32:22.534342212Z", "endTime": "2026-09-22T16:56:24.631597443Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t059ndd697kat3tga063u3hbor3fu8r still exists (deletion dispatched at 2026-09-22T16:32:24Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0010f82a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-5s7lbj2gl95t" to finish deleting: GET https://rp.j2605440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f29ddf1-5f30-4805-8772-bb86c797c186 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1f29ddf1-5f30-4805-8772-bb86c797c186", "name": "1f29ddf1-5f30-4805-8772-bb86c797c186", "status": "Failed", "startTime": "2026-09-22T16:32:22.534342212Z", "endTime": "2026-09-22T16:56:24.631597443Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t059ndd697kat3tga063u3hbor3fu8r still exists (deletion dispatched at 2026-09-22T16:32:24Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:00:34 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0015f4378>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-9wppmrlqtvrm" to finish deleting: GET https://rp.j2605440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/84a79aab-657b-4e08-be66-ba52c724c909 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/84a79aab-657b-4e08-be66-ba52c724c909", "name": "84a79aab-657b-4e08-be66-ba52c724c909", "status": "Failed", "startTime": "2026-09-22T16:14:48.813812977Z", "endTime": "2026-09-22T16:38:51.572108159Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t05ad0e451o6jjr3d5o5rv4piihraka still exists (deletion dispatched at 2026-09-22T16:14:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0015f4378>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-9wppmrlqtvrm" to finish deleting: GET https://rp.j2605440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/84a79aab-657b-4e08-be66-ba52c724c909 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/84a79aab-657b-4e08-be66-ba52c724c909", "name": "84a79aab-657b-4e08-be66-ba52c724c909", "status": "Failed", "startTime": "2026-09-22T16:14:48.813812977Z", "endTime": "2026-09-22T16:38:51.572108159Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t05ad0e451o6jjr3d5o5rv4piihraka still exists (deletion dispatched at 2026-09-22T16:14:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:00:34 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001f861f8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-wvtbxf" in resourcegroup="rg-hs-pre--xzhjszx9k2c7" to finish deleting: GET https://rp.j2605440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/82218643-5493-4a8c-b78b-cdeb6989d0fc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/82218643-5493-4a8c-b78b-cdeb6989d0fc", "name": "82218643-5493-4a8c-b78b-cdeb6989d0fc", "status": "Failed", "startTime": "2026-09-22T16:10:17.188585441Z", "endTime": "2026-09-22T16:34:21.604473698Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t059l0gr7df9ftcbt72ja7n96fahcrt still exists (deletion dispatched at 2026-09-22T16:10:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001f861f8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-hs-pre-wvtbxf" in resourcegroup="rg-hs-pre--xzhjszx9k2c7" to finish deleting: GET https://rp.j2605440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/82218643-5493-4a8c-b78b-cdeb6989d0fc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/82218643-5493-4a8c-b78b-cdeb6989d0fc", "name": "82218643-5493-4a8c-b78b-cdeb6989d0fc", "status": "Failed", "startTime": "2026-09-22T16:10:17.188585441Z", "endTime": "2026-09-22T16:34:21.604473698Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t059l0gr7df9ftcbt72ja7n96fahcrt still exists (deletion dispatched at 2026-09-22T16:10:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:00:34 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000cca6f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-9958km482jsv" to finish deleting: GET https://rp.j2605440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/07560251-b40e-43d9-b428-54fa8fba0e71 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/07560251-b40e-43d9-b428-54fa8fba0e71", "name": "07560251-b40e-43d9-b428-54fa8fba0e71", "status": "Failed", "startTime": "2026-09-22T16:22:28.013921275Z", "endTime": "2026-09-22T16:46:31.609629346Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t059hc0kkjcrvnnkosl8h62gegnbh6e still exists (deletion dispatched at 2026-09-22T16:22:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000cca6f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-9958km482jsv" to finish deleting: GET https://rp.j2605440.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/07560251-b40e-43d9-b428-54fa8fba0e71 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/07560251-b40e-43d9-b428-54fa8fba0e71", "name": "07560251-b40e-43d9-b428-54fa8fba0e71", "status": "Failed", "startTime": "2026-09-22T16:22:28.013921275Z", "endTime": "2026-09-22T16:46:31.609629346Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t059hc0kkjcrvnnkosl8h62gegnbh6e still exists (deletion dispatched at 2026-09-22T16:22:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] FrontendPathMedianLatency fired 15:00:34 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T16:22:37Z Ended: 2026-09-22T16:28:38Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j2605440-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j2605440-svc).
15:00:31	2h 30m 16s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head a65a65d	westus3	alert, e2e	failed	⚠#6756 (open)	5	e2e (4) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:00:31 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001828138>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-hwpggw47v5hm" to finish deleting: GET https://rp.j6892288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63edee93-2a7c-41df-a51b-a7f5f12c3550 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63edee93-2a7c-41df-a51b-a7f5f12c3550", "name": "63edee93-2a7c-41df-a51b-a7f5f12c3550", "status": "Failed", "startTime": "2026-09-22T16:32:51.047517269Z", "endTime": "2026-09-22T16:56:56.822827308Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t05e8a37dph87bq1p0gjquml726f107 still exists (deletion dispatched at 2026-09-22T16:32:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001828138>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-hwpggw47v5hm" to finish deleting: GET https://rp.j6892288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63edee93-2a7c-41df-a51b-a7f5f12c3550 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/63edee93-2a7c-41df-a51b-a7f5f12c3550", "name": "63edee93-2a7c-41df-a51b-a7f5f12c3550", "status": "Failed", "startTime": "2026-09-22T16:32:51.047517269Z", "endTime": "2026-09-22T16:56:56.822827308Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t05e8a37dph87bq1p0gjquml726f107 still exists (deletion dispatched at 2026-09-22T16:32:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:00:31 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000d06288>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-n65mjwh6668r" to finish deleting: GET https://rp.j6892288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5bce1036-3d2d-444f-ad5f-2938541cce95 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5bce1036-3d2d-444f-ad5f-2938541cce95", "name": "5bce1036-3d2d-444f-ad5f-2938541cce95", "status": "Failed", "startTime": "2026-09-22T16:45:55.582699774Z", "endTime": "2026-09-22T17:09:56.885776378Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t05e9l3bms07pg766c7t42iidgiet8o still exists (deletion dispatched at 2026-09-22T16:45:57Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000d06288>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-n65mjwh6668r" to finish deleting: GET https://rp.j6892288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5bce1036-3d2d-444f-ad5f-2938541cce95 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5bce1036-3d2d-444f-ad5f-2938541cce95", "name": "5bce1036-3d2d-444f-ad5f-2938541cce95", "status": "Failed", "startTime": "2026-09-22T16:45:55.582699774Z", "endTime": "2026-09-22T17:09:56.885776378Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t05e9l3bms07pg766c7t42iidgiet8o still exists (deletion dispatched at 2026-09-22T16:45:57Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred failed to gather logs: failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KInternal): Error while getting token : DefaultAzureCredential: failed to acquire a token 15:00:31 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:95]: Timed out after 623.703s. Expected success, but got an error: <fmt.wrapError \| 0xc000c664c0>: failed to gather logs: failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KInternal): Error while getting token : DefaultAzureCredential: failed to acquire a token. Attempted credentials: ClientSecretCredential: unable to resolve an endpoint: server response error: context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret ... fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:95]: Timed out after 623.703s. Expected success, but got an error: <fmt.wrapError \| 0xc000c664c0>: failed to gather logs: failed to execute cluster id query: failed to execute query: failed to execute query: Op(OpQuery): Kind(KInternal): Error while getting token : DefaultAzureCredential: failed to acquire a token. Attempted credentials: ClientSecretCredential: unable to resolve an endpoint: server response error: context deadline exceeded. To troubleshoot, visit https://aka.ms/azsdk/go/identity/troubleshoot#client-secret ... ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 15:00:31 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0016de180>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-8j45b7" in resourcegroup="rg-np-version-upgrade-8j45b7-xfd6p45fzpjm" to finish deleting: GET https://rp.j6892288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef32d17b-db28-47e9-88f8-42656f04a135 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef32d17b-db28-47e9-88f8-42656f04a135", "name": "ef32d17b-db28-47e9-88f8-42656f04a135", "status": "Failed", "startTime": "2026-09-22T16:31:57.110038375Z", "endTime": "2026-09-22T16:56:06.840108207Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t05ebc137s21nadehehjbelf4umr9sv still exists (deletion dispatched at 2026-09-22T16:31:58Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0016de180>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-8j45b7" in resourcegroup="rg-np-version-upgrade-8j45b7-xfd6p45fzpjm" to finish deleting: GET https://rp.j6892288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef32d17b-db28-47e9-88f8-42656f04a135 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef32d17b-db28-47e9-88f8-42656f04a135", "name": "ef32d17b-db28-47e9-88f8-42656f04a135", "status": "Failed", "startTime": "2026-09-22T16:31:57.110038375Z", "endTime": "2026-09-22T16:56:06.840108207Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t05ebc137s21nadehehjbelf4umr9sv still exists (deletion dispatched at 2026-09-22T16:31:58Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] ServiceMemoryDrift fired 15:00:31 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T16:56:25Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j6892288-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-qxgv7", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-qxgv7 (namespace arobit) on cluster ci01-j6892288-svc has averaged 127.1% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
15:00:21	2h 16m 54s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 170f768	westus3	alert, e2e	failed	#6128 (open)	3	e2e (2) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:00:21 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0019a0120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-r6kg2bvtc2sr" to finish deleting: GET https://rp.j0937728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/499e16c4-5cc0-4158-8dd1-44a120f33a5b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/499e16c4-5cc0-4158-8dd1-44a120f33a5b", "name": "499e16c4-5cc0-4158-8dd1-44a120f33a5b", "status": "Failed", "startTime": "2026-09-22T16:34:00.468746528Z", "endTime": "2026-09-22T16:58:07.479939695Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t059hkhmn5f0iq0hki8997j21ldmr5u still exists (deletion dispatched at 2026-09-22T16:34:02Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0019a0120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-r6kg2bvtc2sr" to finish deleting: GET https://rp.j0937728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/499e16c4-5cc0-4158-8dd1-44a120f33a5b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/499e16c4-5cc0-4158-8dd1-44a120f33a5b", "name": "499e16c4-5cc0-4158-8dd1-44a120f33a5b", "status": "Failed", "startTime": "2026-09-22T16:34:00.468746528Z", "endTime": "2026-09-22T16:58:07.479939695Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t059hkhmn5f0iq0hki8997j21ldmr5u still exists (deletion dispatched at 2026-09-22T16:34:02Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 15:00:21 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0011168b8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-69rcqx" in resourcegroup="rg-np-version-upgrade-69rcqx-t7vvxb4m82c2" to finish deleting: GET https://rp.j0937728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ff253b2-25f8-4257-bbb3-405b53cbb8c3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ff253b2-25f8-4257-bbb3-405b53cbb8c3", "name": "1ff253b2-25f8-4257-bbb3-405b53cbb8c3", "status": "Failed", "startTime": "2026-09-22T16:21:27.793968638Z", "endTime": "2026-09-22T16:45:27.454994019Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t059ko8q63a62o47obuopvif25t4gq7 still exists (deletion dispatched at 2026-09-22T16:21:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0011168b8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-69rcqx" in resourcegroup="rg-np-version-upgrade-69rcqx-t7vvxb4m82c2" to finish deleting: GET https://rp.j0937728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ff253b2-25f8-4257-bbb3-405b53cbb8c3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1ff253b2-25f8-4257-bbb3-405b53cbb8c3", "name": "1ff253b2-25f8-4257-bbb3-405b53cbb8c3", "status": "Failed", "startTime": "2026-09-22T16:21:27.793968638Z", "endTime": "2026-09-22T16:45:27.454994019Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t059ko8q63a62o47obuopvif25t4gq7 still exists (deletion dispatched at 2026-09-22T16:21:30Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] ServiceMemoryDrift fired 15:00:21 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T17:03:42Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j0937728-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-nbw4j", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-nbw4j (namespace arobit) on cluster ci01-j0937728-svc has averaged 126% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
14:48:11	2h 3m 30s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 115a341	canadacentral	alert	failed	★#6821 (open)	2	alert (2) alert [svc] FrontendPathMedianLatency fired 14:48:11 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T16:07:49Z Ended: 2026-09-22T16:18:48Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j5857792-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning" Description: The median (p50) of frontend request latency for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} has exceeded 250ms over the past 30 minutes (cluster ci01-j5857792-svc). alert [svc] userJourneyFrontendLatency6h30m fired 14:48:11 UTC · failed at alert · test [aro-hcp-observability] [svc] alert userJourneyFrontendLatency6h30m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T16:13:02Z Severity: Sev3 Labels: alertname="userJourneyFrontendLatency6h30m", cluster="ci01-j5857792-svc", component="slo", long_window="6h", region="canadacentral", severity="3", short_window="30m", slo="frontend-latency" Description: More than 6% of Frontend requests on cluster ci01-j5857792-svc exceeded 1s on both the 6h and 30m windows (6x burn of the 1% latency budget).
14:46:39	2h 10m 11s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 23179a5 · merge 23179a5	canadacentral	n/a	passed	#7066 (merged)	0	n/a
14:46:04	2h 1m 20s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head a66c056 · merge a66c056	centralus	alert	failed	#7079 (merged)	1	alert (1) alert [svc] BackendControllerQueueDepthHigh fired 14:46:04 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T15:47:19Z Ended: 2026-09-22T16:02:16Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5304576-svc", component="backend", name="clusterresources", region="centralus", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
14:44:25	2h 24m 57s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c530dc6	westus3	alert, e2e	failed	#6364 (open)	5	e2e (4) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 14:44:25 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error: <errors.joinError \| 0xc00109e318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-skip-minor-xfn7hz" in resourcegroup="rg-np-skip-minor-xfn7hz-fxl2864zknf9" to finish deleting: GET https://rp.j0209536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6ca80c75-8e81-4c60-bd28-79f67fe671f4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6ca80c75-8e81-4c60-bd28-79f67fe671f4", "name": "6ca80c75-8e81-4c60-bd28-79f67fe671f4", "status": "Failed", "startTime": "2026-09-22T16:23:52.521516326Z", "endTime": "2026-09-22T16:47:53.92980352Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t055v8ffvgh7hoqu3ka4e48gk1k5rmp still exists (deletion dispatched at 2026-09-22T16:23:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error: <errors.joinError \| 0xc00109e318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-skip-minor-xfn7hz" in resourcegroup="rg-np-skip-minor-xfn7hz-fxl2864zknf9" to finish deleting: GET https://rp.j0209536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6ca80c75-8e81-4c60-bd28-79f67fe671f4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6ca80c75-8e81-4c60-bd28-79f67fe671f4", "name": "6ca80c75-8e81-4c60-bd28-79f67fe671f4", "status": "Failed", "startTime": "2026-09-22T16:23:52.521516326Z", "endTime": "2026-09-22T16:47:53.92980352Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t055v8ffvgh7hoqu3ka4e48gk1k5rmp still exists (deletion dispatched at 2026-09-22T16:23:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 14:44:25 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error: <errors.joinError \| 0xc001c8a1e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-qggbmvr2rwbm" to finish deleting: GET https://rp.j0209536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3cc3a241-4d56-45df-ab75-ee83af76ddc0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3cc3a241-4d56-45df-ab75-ee83af76ddc0", "name": "3cc3a241-4d56-45df-ab75-ee83af76ddc0", "status": "Failed", "startTime": "2026-09-22T16:26:02.874087694Z", "endTime": "2026-09-22T16:50:03.912563121Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0563ura2ok3oborcab1tonlj3tu42b still exists (deletion dispatched at 2026-09-22T16:26:04Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error: <errors.joinError \| 0xc001c8a1e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-qggbmvr2rwbm" to finish deleting: GET https://rp.j0209536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3cc3a241-4d56-45df-ab75-ee83af76ddc0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3cc3a241-4d56-45df-ab75-ee83af76ddc0", "name": "3cc3a241-4d56-45df-ab75-ee83af76ddc0", "status": "Failed", "startTime": "2026-09-22T16:26:02.874087694Z", "endTime": "2026-09-22T16:50:03.912563121Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0563ura2ok3oborcab1tonlj3tu42b still exists (deletion dispatched at 2026-09-22T16:26:04Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist 14:44:25 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.004s. Certificate should be issued by Let's Encrypt staging via ACME DNS-01 Expected success, but got an error: <errors.errorString \| 0xc002204a80>: certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.004s. Certificate should be issued by Let's Encrypt staging via ACME DNS-01 Expected success, but got an error: <errors.errorString \| 0xc002204a80>: certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist ... ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 14:44:25 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error: <errors.joinError \| 0xc001886318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-spgcj2kh2h5v" to finish deleting: GET https://rp.j0209536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bcde3a95-b9f7-4460-9746-02c9c95942bd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bcde3a95-b9f7-4460-9746-02c9c95942bd", "name": "bcde3a95-b9f7-4460-9746-02c9c95942bd", "status": "Failed", "startTime": "2026-09-22T16:12:37.093658582Z", "endTime": "2026-09-22T16:36:43.921137452Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0566nivsb9spt22bvvibas1l9sn9op still exists (deletion dispatched at 2026-09-22T16:12:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error: <errors.joinError \| 0xc001886318>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-spgcj2kh2h5v" to finish deleting: GET https://rp.j0209536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bcde3a95-b9f7-4460-9746-02c9c95942bd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bcde3a95-b9f7-4460-9746-02c9c95942bd", "name": "bcde3a95-b9f7-4460-9746-02c9c95942bd", "status": "Failed", "startTime": "2026-09-22T16:12:37.093658582Z", "endTime": "2026-09-22T16:36:43.921137452Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0566nivsb9spt22bvvibas1l9sn9op still exists (deletion dispatched at 2026-09-22T16:12:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] ServiceMemoryDrift fired 14:44:25 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T16:41:53Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j0209536-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-pv857", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-pv857 (namespace arobit) on cluster ci01-j0209536-svc has averaged 126.4% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
14:42:34	2h 22m 8s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 5126a8a	centralus	e2e	failed	#7054 (open)	2	e2e (2) certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist 14:42:34 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.001s. Certificate should be issued by Let's Encrypt staging via ACME DNS-01 Expected success, but got an error: <errors.errorString \| 0xc00176c920>: certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.001s. Certificate should be issued by Let's Encrypt staging via ACME DNS-01 Expected success, but got an error: <errors.errorString \| 0xc00176c920>: certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist ... ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 14:42:34 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000b962a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-cvbhkw49wbpm" to finish deleting: GET https://rp.j1058944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f877ac3b-3a58-48db-ab63-47dc469a8550 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f877ac3b-3a58-48db-ab63-47dc469a8550", "name": "f877ac3b-3a58-48db-ab63-47dc469a8550", "status": "Failed", "startTime": "2026-09-22T16:22:27.181566391Z", "endTime": "2026-09-22T16:46:34.108010104Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t053nni8oc5l9492pcvasv7p0llgkk9 still exists (deletion dispatched at 2026-09-22T16:22:28Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000b962a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-cvbhkw49wbpm" to finish deleting: GET https://rp.j1058944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f877ac3b-3a58-48db-ab63-47dc469a8550 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f877ac3b-3a58-48db-ab63-47dc469a8550", "name": "f877ac3b-3a58-48db-ab63-47dc469a8550", "status": "Failed", "startTime": "2026-09-22T16:22:27.181566391Z", "endTime": "2026-09-22T16:46:34.108010104Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t053nni8oc5l9492pcvasv7p0llgkk9 still exists (deletion dispatched at 2026-09-22T16:22:28Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
14:38:17	2h 16m 44s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 6cb0492 · merge 6cb0492	canadacentral	alert, e2e	failed	#7020 (merged)	2	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 14:38:17 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a88078>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-qdkndq" in resourcegroup="rg-zstream-upgrade-4-20-mgkkt9xlp7cv" to finish deleting: GET https://rp.j3467520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e5070f36-fffe-45fb-8025-aaa6534d21ee -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e5070f36-fffe-45fb-8025-aaa6534d21ee", "name": "e5070f36-fffe-45fb-8025-aaa6534d21ee", "status": "Failed", "startTime": "2026-09-22T16:00:42.037171162Z", "endTime": "2026-09-22T16:24:46.401073716Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t052krp2n6diohfm8enfagipjapnjd7 still exists (deletion dispatched at 2026-09-22T16:01:06Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a88078>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-qdkndq" in resourcegroup="rg-zstream-upgrade-4-20-mgkkt9xlp7cv" to finish deleting: GET https://rp.j3467520.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e5070f36-fffe-45fb-8025-aaa6534d21ee -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e5070f36-fffe-45fb-8025-aaa6534d21ee", "name": "e5070f36-fffe-45fb-8025-aaa6534d21ee", "status": "Failed", "startTime": "2026-09-22T16:00:42.037171162Z", "endTime": "2026-09-22T16:24:46.401073716Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t052krp2n6diohfm8enfagipjapnjd7 still exists (deletion dispatched at 2026-09-22T16:01:06Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] FrontendPathMedianLatency fired 14:38:17 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T16:09:00Z Ended: 2026-09-22T16:14:59Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j3467520-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3467520-svc).
14:16:24	1h 54m 55s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 901730f	canadacentral	alert, e2e	failed	#7071 (open)	2	e2e (1) ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 14:16:24 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-55rf9x" for cluster "cluster-candidate-5-0-55rf9x" Unexpected error: <fmt.wrapError \| 0xc001027d20>: failed to create NodePool np-55rf9x: failed waiting for nodepool="np-55rf9x" for cluster "cluster-candidate-5-0-55rf9x" in resourcegroup="rg-candidate-5-0-tv9lmfcf58jh" to finish creating: GET https://rp.j3849216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e6c15d22-5a6f-4305-86d0-76dd664e2eeb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e6c15d22-5a6f-4305-86d0-76dd664e2eeb", "name": "e6c15d22-5a6f-4305-86d0-76dd664e2eeb", "status": "Failed", "startTime": "2026-09-22T15:06:40.249228102Z", "endTime": "2026-09-22T15:25:48.538295529Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-55rf9x" for cluster "cluster-candidate-5-0-55rf9x" Unexpected error: <fmt.wrapError \| 0xc001027d20>: failed to create NodePool np-55rf9x: failed waiting for nodepool="np-55rf9x" for cluster "cluster-candidate-5-0-55rf9x" in resourcegroup="rg-candidate-5-0-tv9lmfcf58jh" to finish creating: GET https://rp.j3849216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e6c15d22-5a6f-4305-86d0-76dd664e2eeb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e6c15d22-5a6f-4305-86d0-76dd664e2eeb", "name": "e6c15d22-5a6f-4305-86d0-76dd664e2eeb", "status": "Failed", "startTime": "2026-09-22T15:06:40.249228102Z", "endTime": "2026-09-22T15:25:48.538295529Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] FrontendPathMedianLatency fired 14:16:24 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T15:33:28Z Ended: 2026-09-22T15:39:28Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j3849216-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3849216-svc).
14:11:35	2h 3m 56s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 911c458	centralus	e2e	failed	#6662 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 14:11:35 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001ac2060>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-x9wsbz" in resourcegroup="rg-zstream-upgrade-4-20-fssqcz7gkd95" to finish deleting: GET https://rp.j4023552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8523fbca-d533-47f4-9ac4-f0f2cfb1d97e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8523fbca-d533-47f4-9ac4-f0f2cfb1d97e", "name": "8523fbca-d533-47f4-9ac4-f0f2cfb1d97e", "status": "Failed", "startTime": "2026-09-22T15:30:10.282474765Z", "endTime": "2026-09-22T15:54:15.386999815Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t04kdenf2urnkc2mp70smb8379c9eh1 still exists (deletion dispatched at 2026-09-22T15:30:14Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001ac2060>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-x9wsbz" in resourcegroup="rg-zstream-upgrade-4-20-fssqcz7gkd95" to finish deleting: GET https://rp.j4023552.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8523fbca-d533-47f4-9ac4-f0f2cfb1d97e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8523fbca-d533-47f4-9ac4-f0f2cfb1d97e", "name": "8523fbca-d533-47f4-9ac4-f0f2cfb1d97e", "status": "Failed", "startTime": "2026-09-22T15:30:10.282474765Z", "endTime": "2026-09-22T15:54:15.386999815Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t04kdenf2urnkc2mp70smb8379c9eh1 still exists (deletion dispatched at 2026-09-22T15:30:14Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
14:08:23	2h 17m 50s	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head df1f8c9 · merge a1bb525	centralus	alert, e2e	failed	#7072 (merged)	2	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 14:08:23 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00136a120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-z2xhl2smhl95" to finish deleting: GET https://rp.j2940800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/020b2ad2-90c1-42cf-954f-9c42af86b89a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/020b2ad2-90c1-42cf-954f-9c42af86b89a", "name": "020b2ad2-90c1-42cf-954f-9c42af86b89a", "status": "Failed", "startTime": "2026-09-22T15:45:07.411300012Z", "endTime": "2026-09-22T16:09:12.675827628Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t04j35581secir5jt8vd82f1nps1e1h still exists (deletion dispatched at 2026-09-22T15:45:09Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00136a120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-z2xhl2smhl95" to finish deleting: GET https://rp.j2940800.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/020b2ad2-90c1-42cf-954f-9c42af86b89a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/020b2ad2-90c1-42cf-954f-9c42af86b89a", "name": "020b2ad2-90c1-42cf-954f-9c42af86b89a", "status": "Failed", "startTime": "2026-09-22T15:45:07.411300012Z", "endTime": "2026-09-22T16:09:12.675827628Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t04j35581secir5jt8vd82f1nps1e1h still exists (deletion dispatched at 2026-09-22T15:45:09Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] ServiceMemoryDrift fired 14:08:23 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T16:03:28Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j2940800-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-px2fd", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-px2fd (namespace arobit) on cluster ci01-j2940800-svc has averaged 126.3% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
13:59:53	2h 14m 26s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 87e40a0	westus3	alert, e2e	failed	#6922 (open)	3	e2e (2) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 13:59:53 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00113c168>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-kgt4dp" in resourcegroup="rg-cp-ystream-upgrade-4-22-4927vw69rw8m" to finish deleting: GET https://rp.j8035328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e1e0cb39-53d3-4858-ad0a-0f3a35aac6ba -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e1e0cb39-53d3-4858-ad0a-0f3a35aac6ba", "name": "e1e0cb39-53d3-4858-ad0a-0f3a35aac6ba", "status": "Failed", "startTime": "2026-09-22T15:13:19.013850014Z", "endTime": "2026-09-22T15:37:24.201740062Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t04dltg78ddmiii1fbhn53kmu5p1uov still exists (deletion dispatched at 2026-09-22T15:13:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00113c168>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-kgt4dp" in resourcegroup="rg-cp-ystream-upgrade-4-22-4927vw69rw8m" to finish deleting: GET https://rp.j8035328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e1e0cb39-53d3-4858-ad0a-0f3a35aac6ba -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e1e0cb39-53d3-4858-ad0a-0f3a35aac6ba", "name": "e1e0cb39-53d3-4858-ad0a-0f3a35aac6ba", "status": "Failed", "startTime": "2026-09-22T15:13:19.013850014Z", "endTime": "2026-09-22T15:37:24.201740062Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t04dltg78ddmiii1fbhn53kmu5p1uov still exists (deletion dispatched at 2026-09-22T15:13:20Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 13:59:53 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a120a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-ff54k9w27kdm" to finish deleting: GET https://rp.j8035328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/71c3d3ae-fcf9-46a8-a5ab-5481a9f5283e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/71c3d3ae-fcf9-46a8-a5ab-5481a9f5283e", "name": "71c3d3ae-fcf9-46a8-a5ab-5481a9f5283e", "status": "Failed", "startTime": "2026-09-22T15:31:35.352639854Z", "endTime": "2026-09-22T15:55:44.858066336Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t04dmjhorlcc8rbknfj93atobs76t0i still exists (deletion dispatched at 2026-09-22T15:31:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a120a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-ff54k9w27kdm" to finish deleting: GET https://rp.j8035328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/71c3d3ae-fcf9-46a8-a5ab-5481a9f5283e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/71c3d3ae-fcf9-46a8-a5ab-5481a9f5283e", "name": "71c3d3ae-fcf9-46a8-a5ab-5481a9f5283e", "status": "Failed", "startTime": "2026-09-22T15:31:35.352639854Z", "endTime": "2026-09-22T15:55:44.858066336Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t04dmjhorlcc8rbknfj93atobs76t0i still exists (deletion dispatched at 2026-09-22T15:31:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] ServiceMemoryDrift fired 13:59:53 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T15:54:51Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j8035328-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-xf56d", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-xf56d (namespace arobit) on cluster ci01-j8035328-svc has averaged 124% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
13:36:14	1h 58m 56s	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head c12349d · merge 53e6eda	westus3	e2e	failed	⚠#7077 (merged)	1	e2e (1) VerifySimpleWebApp failed: DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded 13:36:14 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update Unexpected error: <errors.joinError \| 0xc001514030>: VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:203]: failed to verify cluster workloads and pods after autoscaling update Unexpected error: <errors.joinError \| 0xc001514030>: VerifySimpleWebApp failed: DNS for route host agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-ctpxj.apps.aro.v4m3f6y4q7z0t3b.chi6.j5582976.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded ... occurred
13:22:45	2h 10m 4s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 604bf18	centralus	alert, e2e	failed	#7001 (open)	3	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 13:22:45 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0012680c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-kzfns6p8hnnz" to finish deleting: GET https://rp.j0529536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/357dbfd6-1ecc-49c9-8451-726c0525914c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/357dbfd6-1ecc-49c9-8451-726c0525914c", "name": "357dbfd6-1ecc-49c9-8451-726c0525914c", "status": "Failed", "startTime": "2026-09-22T14:51:53.865556884Z", "endTime": "2026-09-22T15:16:02.876997439Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t03qlefes7qqu9ndhbba63al41fsd54 still exists (deletion dispatched at 2026-09-22T14:51:55Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0012680c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-kzfns6p8hnnz" to finish deleting: GET https://rp.j0529536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/357dbfd6-1ecc-49c9-8451-726c0525914c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/357dbfd6-1ecc-49c9-8451-726c0525914c", "name": "357dbfd6-1ecc-49c9-8451-726c0525914c", "status": "Failed", "startTime": "2026-09-22T14:51:53.865556884Z", "endTime": "2026-09-22T15:16:02.876997439Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t03qlefes7qqu9ndhbba63al41fsd54 still exists (deletion dispatched at 2026-09-22T14:51:55Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [svc] FrontendPathMedianLatency fired 13:22:45 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T14:34:27Z Ended: 2026-09-22T14:47:29Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j0529536-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0529536-svc). alert [svc] ServiceMemoryDrift fired 13:22:45 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T15:19:33Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j0529536-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-zt76m", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-zt76m (namespace arobit) on cluster ci01-j0529536-svc has averaged 127% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
13:12:15	2h 16m 58s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head f8180cd	westus3	alert, e2e	failed	#6688 (open)	4	e2e (2) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 13:12:15 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0011701e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-z58j7fdlqz95" to finish deleting: GET https://rp.j5833600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/684e8ee0-2c18-411e-acc4-32c72b510d88 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/684e8ee0-2c18-411e-acc4-32c72b510d88", "name": "684e8ee0-2c18-411e-acc4-32c72b510d88", "status": "Failed", "startTime": "2026-09-22T14:47:35.770349631Z", "endTime": "2026-09-22T15:11:39.446675519Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t03nuv85l4gu6j6jj010pj9d9bc1a7p still exists (deletion dispatched at 2026-09-22T14:47:36Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0011701e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-z58j7fdlqz95" to finish deleting: GET https://rp.j5833600.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/684e8ee0-2c18-411e-acc4-32c72b510d88 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/684e8ee0-2c18-411e-acc4-32c72b510d88", "name": "684e8ee0-2c18-411e-acc4-32c72b510d88", "status": "Failed", "startTime": "2026-09-22T14:47:35.770349631Z", "endTime": "2026-09-22T15:11:39.446675519Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t03nuv85l4gu6j6jj010pj9d9bc1a7p still exists (deletion dispatched at 2026-09-22T14:47:36Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred client rate limiter Wait returned an error: context deadline exceeded 13:12:15 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation Unexpected error: <errors.joinError \| 0xc0025d8000>: context deadline exceeded failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:251]: failed to verify cluster operators are available after external auth config creation Unexpected error: <errors.joinError \| 0xc0025d8000>: context deadline exceeded failed to list ClusterOperators: client rate limiter Wait returned an error: context deadline exceeded ... occurred alert (2) alert [svc] ServiceMemoryDrift fired 13:12:15 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T15:07:54Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j5833600-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-cn6gk", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-cn6gk (namespace arobit) on cluster ci01-j5833600-svc has averaged 125.3% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml. alert [svc] BackendControllerQueueDepthHigh fired 13:12:15 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T14:26:28Z Ended: 2026-09-22T14:42:27Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j5833600-svc", component="backend", name="datadump", region="westus3", severity="warning" Description: Backend controller workqueue datadump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
13:02:00	2h 16m 10s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 8ae5d54 · merge 8ae5d54	canadacentral	e2e	failed	#7067 (merged)	1	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 13:02:00 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000b0e6c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-psj6bqhkk247" to finish deleting: GET https://rp.j6961408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b1507794-ddcc-4272-8150-61c8e2db2951 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b1507794-ddcc-4272-8150-61c8e2db2951", "name": "b1507794-ddcc-4272-8150-61c8e2db2951", "status": "Failed", "startTime": "2026-09-22T14:37:48.593192113Z", "endTime": "2026-09-22T15:01:49.089274756Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t03iul7c5s89v3g3tp3eoc5uotn7s1q still exists (deletion dispatched at 2026-09-22T14:37:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000b0e6c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-psj6bqhkk247" to finish deleting: GET https://rp.j6961408.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b1507794-ddcc-4272-8150-61c8e2db2951 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b1507794-ddcc-4272-8150-61c8e2db2951", "name": "b1507794-ddcc-4272-8150-61c8e2db2951", "status": "Failed", "startTime": "2026-09-22T14:37:48.593192113Z", "endTime": "2026-09-22T15:01:49.089274756Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t03iul7c5s89v3g3tp3eoc5uotn7s1q still exists (deletion dispatched at 2026-09-22T14:37:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
13:01:50	51m 36s	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head eca3647 · merge 23179a5	westus3	provision	failed	⚠#7066 (merged)	1	provision (1) failed to roll out Helm release: failed pre-install: resource Job/prometheus/prometheus-admission-create not ready. status: Failed, message: Job Failed. failed: 1/1 13:01:50 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/prometheus · suite templatize-pipeline Show raw failure time=2026-09-22T13:40:24.327Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus description="Step prometheus\n Kind: Helm\n" time=2026-09-22T13:40:25.067Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus values="map[environment:arohcpci01 kube-prometheus-stack:map[alertmanager:map[enabled:false] cleanPrometheusOperatorObjectNames:true coreDns:map[enabled:false] crds:map[enabled:true] fullnameOverride:prometheus global:map[rbac:map[create:true]] grafana:map[enabled:false] kube-state-metrics:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:kubernetes/kube-state-metrics sha:sha256:9e63919b9dde3a44483a565c55744abf9eab54314586f01a631a42b1104d5f2a] metricLabelsAllowlist:[nodes=[node.kubernetes.io/instance-type,kubernetes.azure.com/agentpool]]] kubeApiServer:map[enabled:false] kubeControllerManager:map[enabled:false] kubeDns:map[enabled:false] kubeEtcd:map[enabled:false] kubeProxy:map[enabled:false] kubeScheduler:map[enabled:false] kubeStateMetrics:map[enabled:true] kubelet:map[enabled:false] nodeExporter:map[enabled:false] prometheus:map[enabled:false] prometheusOperator:map[admissionWebhooks:map[patch:map[enabled:true image:map[registry:arohcpsvcdev.azurecr.io repository:k8s-cache/ingress-nginx/kube-webhook-certgen sha:01038e7de14b78d702d2849c3aad72fd25903c4765af63cf16aa3398f5d5f2dd]]] affinity:map[nodeAffinity:map[requiredDuringSchedulingIgnoredDuringExecution:map[nodeSelectorTerms:[map[matchExpressions:[map[key:aro-hcp.azure.com/role operator:In values:[infra]]]]]]]] enabled:true fullnameOverride: image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-operator sha:be4e1b8bf4b8cc868cfa993818ed94e2536adc15713fc64942ce6a47d1bbabb9] prometheusConfigReloader:map[image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus-config-reloader sha:b840d65d98ba99f4d62329db721714b36ec23d7a0c8721d1fdb4fc839f97cf76]] thanosImage:map[registry:arohcpsvcdev.azurecr.io repository:thanos/thanos sha:b567818fe608067eb0f1d7c2c4fe361e7ad83c8a256234c97685f1d0bf670cc8 tag:v0.41.0] tolerations:[map[effect:NoSchedule key:infra operator:Equal value:true]]]] prometheus:map[serviceAccount:map[managedIdentity:d7ccda51-6374-4c36-847f-8c68f35d13dc]] prometheusSpec:map[externalLabels:map[cluster:ci01-j6088704-svc environment:ci01 region:westus3] hcpRemoteWriteUrl:NONE image:map[registry:mcr.microsoft.com/oss/v2 repository:prometheus/prometheus sha:92fa8f5e91a0e02e0107c346f6c66657738dbf394946e52c208150fd498554bd] maximumStartupDurationSeconds:360 remoteWriteUrl:https://msprom-westus3-ci01-j6088704-svc-ybo2.westus3-1.metrics.ingest.monitor.azure.com/dataCollectionRules/dcr-7b09a45e774144ada99a9dc87a23c02f/streams/Microsoft-PrometheusMetrics/api/v1/write?api-version=2023-04-24 replicas:2 resources:map[limits:map[memory:NONE] requests:map[cpu:NONE memory:NONE]] shards:1 version: zoneCount:3]]" time=2026-09-22T13:40:25.067Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-09-22T13:40:25.331Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus namespace=prometheus time=2026-09-22T13:40:25.636Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus dryRun=false time=2026-09-22T13:40:25.636Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-09-22T13:40:25.815Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus time=2026-09-22T13:46:35.923Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="failed pre-install: resource Job/prometheus/prometheus-admission-create not ready. status: Failed, message: Job Failed. failed: 1/1" time=2026-09-22T13:46:35.923Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=prometheus err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: failed pre-install: resource Job/prometheus/prometheus-admission-create not ready. status: Failed, message: Job Failed. failed: 1/1"
13:01:46	2h 3m 58s	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 6c5cfd0 · merge a0bf660	centralus	alert, e2e	failed	#6503 (merged)	2	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 13:01:46 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00199c120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-gvrrfd7mjt5j" to finish deleting: GET https://rp.j1247360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5c52fcaf-7a1b-409f-8d01-0b1f6fe3e2eb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5c52fcaf-7a1b-409f-8d01-0b1f6fe3e2eb", "name": "5c52fcaf-7a1b-409f-8d01-0b1f6fe3e2eb", "status": "Failed", "startTime": "2026-09-22T14:22:32.067338295Z", "endTime": "2026-09-22T14:46:39.174252031Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t03iv79memg0f0b55eshd59cvgqg187 still exists (deletion dispatched at 2026-09-22T14:22:33Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00199c120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-gvrrfd7mjt5j" to finish deleting: GET https://rp.j1247360.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5c52fcaf-7a1b-409f-8d01-0b1f6fe3e2eb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5c52fcaf-7a1b-409f-8d01-0b1f6fe3e2eb", "name": "5c52fcaf-7a1b-409f-8d01-0b1f6fe3e2eb", "status": "Failed", "startTime": "2026-09-22T14:22:32.067338295Z", "endTime": "2026-09-22T14:46:39.174252031Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t03iv79memg0f0b55eshd59cvgqg187 still exists (deletion dispatched at 2026-09-22T14:22:33Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] FrontendPathMedianLatency fired 13:01:46 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T13:56:14Z Ended: 2026-09-22T14:30:13Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j1247360-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1247360-svc).
12:46:33	2h 9m 50s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 01ae0b3	westus3	alert, e2e	failed	⚠#6889 (open)	63	e2e (61) ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to cordon, drain, and uncordon a node in an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000fa07e0>: failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-9vb62w25cp44" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8709ce20-9ece-4f8c-adc6-a03474e00ed8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8709ce20-9ece-4f8c-adc6-a03474e00ed8", "name": "8709ce20-9ece-4f8c-adc6-a03474e00ed8", "status": "Failed", "startTime": "2026-09-22T13:32:48.411427372Z", "endTime": "2026-09-22T14:01:54.964813171Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000fa07e0>: failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-9vb62w25cp44" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8709ce20-9ece-4f8c-adc6-a03474e00ed8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8709ce20-9ece-4f8c-adc6-a03474e00ed8", "name": "8709ce20-9ece-4f8c-adc6-a03474e00ed8", "status": "Failed", "startTime": "2026-09-22T13:32:48.411427372Z", "endTime": "2026-09-22T14:01:54.964813171Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred cluster entered terminal Failed state after role assignment deployment 12:46:33 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s. cluster should eventually succeed after role assignments are created The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with: cluster entered terminal Failed state after role assignment deployment Expected <armredhatopenshifthcp.ProvisioningState>: ... not to equal <armredhatopenshifthcp.ProvisioningState>: ... ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-tpxqzt" at install version 4.21 Unexpected error: <fmt.wrapError \| 0xc000548ca0>: failed to create HCP cluster cp-ystream-upgrade-4-22-tpxqzt: failed waiting for cluster="cp-ystream-upgrade-4-22-tpxqzt" in resourcegroup="rg-cp-ystream-upgrade-4-22-kzmrc9msf5wb" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c339ee69-a414-453c-a869-0268b31874e6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c339ee69-a414-453c-a869-0268b31874e6", "name": "c339ee69-a414-453c-a869-0268b31874e6", "status": "Failed", "startTime": "2026-09-22T13:32:20.2901057Z", "endTime": "2026-09-22T14:01:24.891016711Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-tpxqzt" at install version 4.21 Unexpected error: <fmt.wrapError \| 0xc000548ca0>: failed to create HCP cluster cp-ystream-upgrade-4-22-tpxqzt: failed waiting for cluster="cp-ystream-upgrade-4-22-tpxqzt" in resourcegroup="rg-cp-ystream-upgrade-4-22-kzmrc9msf5wb" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c339ee69-a414-453c-a869-0268b31874e6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c339ee69-a414-453c-a869-0268b31874e6", "name": "c339ee69-a414-453c-a869-0268b31874e6", "status": "Failed", "startTime": "2026-09-22T13:32:20.2901057Z", "endTime": "2026-09-22T14:01:24.891016711Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc001034880>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-kh9mclv7h744" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/957231f3-0c5e-474f-967b-9b836b1bcff8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/957231f3-0c5e-474f-967b-9b836b1bcff8", "name": "957231f3-0c5e-474f-967b-9b836b1bcff8", "status": "Failed", "startTime": "2026-09-22T13:34:03.567742056Z", "endTime": "2026-09-22T14:03:04.97548831Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc001034880>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-kh9mclv7h744" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/957231f3-0c5e-474f-967b-9b836b1bcff8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/957231f3-0c5e-474f-967b-9b836b1bcff8", "name": "957231f3-0c5e-474f-967b-9b836b1bcff8", "status": "Failed", "startTime": "2026-09-22T13:34:03.567742056Z", "endTime": "2026-09-22T14:03:04.97548831Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc000f266c0>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-k2p6znpk7xx7" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef41379b-6f3e-4285-8db4-e555e4ff61f6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef41379b-6f3e-4285-8db4-e555e4ff61f6", "name": "ef41379b-6f3e-4285-8db4-e555e4ff61f6", "status": "Failed", "startTime": "2026-09-22T13:32:27.09771924Z", "endTime": "2026-09-22T14:01:34.874183273Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc000f266c0>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-k2p6znpk7xx7" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef41379b-6f3e-4285-8db4-e555e4ff61f6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef41379b-6f3e-4285-8db4-e555e4ff61f6", "name": "ef41379b-6f3e-4285-8db4-e555e4ff61f6", "status": "Failed", "startTime": "2026-09-22T13:32:27.09771924Z", "endTime": "2026-09-22T14:01:34.874183273Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes) Unexpected error: <exported.ResponseError \| 0xc001812570>: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3eb8ee2-faee-44d1-b236-05c68194f7e8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3eb8ee2-faee-44d1-b236-05c68194f7e8", "name": "c3eb8ee2-faee-44d1-b236-05c68194f7e8", "status": "Failed", "startTime": "2026-09-22T13:32:46.407843603Z", "endTime": "2026-09-22T14:01:54.958239729Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes) Unexpected error: <exported.ResponseError \| 0xc001812570>: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3eb8ee2-faee-44d1-b236-05c68194f7e8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3eb8ee2-faee-44d1-b236-05c68194f7e8", "name": "c3eb8ee2-faee-44d1-b236-05c68194f7e8", "status": "Failed", "startTime": "2026-09-22T13:32:46.407843603Z", "endTime": "2026-09-22T14:01:54.958239729Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc000a60ba0>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-862l4sm6zmr9" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c11a3871-fc26-4435-a797-b908d4806163 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c11a3871-fc26-4435-a797-b908d4806163", "name": "c11a3871-fc26-4435-a797-b908d4806163", "status": "Failed", "startTime": "2026-09-22T13:31:51.078510413Z", "endTime": "2026-09-22T14:00:54.733017321Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc000a60ba0>: failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-862l4sm6zmr9" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c11a3871-fc26-4435-a797-b908d4806163 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c11a3871-fc26-4435-a797-b908d4806163", "name": "c11a3871-fc26-4435-a797-b908d4806163", "status": "Failed", "startTime": "2026-09-22T13:31:51.078510413Z", "endTime": "2026-09-22T14:00:54.733017321Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc000ed48e0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-fbhlhhg249qs" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42374bce-7c8e-4634-9787-3fba14ef004b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42374bce-7c8e-4634-9787-3fba14ef004b", "name": "42374bce-7c8e-4634-9787-3fba14ef004b", "status": "Failed", "startTime": "2026-09-22T14:08:04.562790307Z", "endTime": "2026-09-22T14:37:08.529968518Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc000ed48e0>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-fbhlhhg249qs" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42374bce-7c8e-4634-9787-3fba14ef004b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/42374bce-7c8e-4634-9787-3fba14ef004b", "name": "42374bce-7c8e-4634-9787-3fba14ef004b", "status": "Failed", "startTime": "2026-09-22T14:08:04.562790307Z", "endTime": "2026-09-22T14:37:08.529968518Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-kbjcpq" with version 4.21.33 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000f728e0>: failed to create HCP cluster cluster-zstream-4-21-kbjcpq: failed waiting for cluster="cluster-zstream-4-21-kbjcpq" in resourcegroup="rg-zstream-upgrade-4-21-ftfjp89m25kk" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d11e0bc-f5d8-4182-a6e1-387ef6e21ad9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d11e0bc-f5d8-4182-a6e1-387ef6e21ad9", "name": "3d11e0bc-f5d8-4182-a6e1-387ef6e21ad9", "status": "Failed", "startTime": "2026-09-22T13:32:22.061338875Z", "endTime": "2026-09-22T14:01:24.909726517Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-kbjcpq" with version 4.21.33 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000f728e0>: failed to create HCP cluster cluster-zstream-4-21-kbjcpq: failed waiting for cluster="cluster-zstream-4-21-kbjcpq" in resourcegroup="rg-zstream-upgrade-4-21-ftfjp89m25kk" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d11e0bc-f5d8-4182-a6e1-387ef6e21ad9 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3d11e0bc-f5d8-4182-a6e1-387ef6e21ad9", "name": "3d11e0bc-f5d8-4182-a6e1-387ef6e21ad9", "status": "Failed", "startTime": "2026-09-22T13:32:22.061338875Z", "endTime": "2026-09-22T14:01:24.909726517Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred scheduling requested CPU must be positive 12:46:33 UTC · failed at e2e · test Fleet should have scheduling data for ready management clusters · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.001s. scheduling data was not populated in time The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:156 with: stamp 2 scheduling requested CPU must be positive Expected <bool>: ... to be false fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:162]: Timed out after 900.001s. scheduling data was not populated in time The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:156 with: stamp 2 scheduling requested CPU must be positive Expected <bool>: ... to be false ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000bb8c80>: failed to create HCP cluster np-dg-minor-8jkpl9: failed waiting for cluster="np-dg-minor-8jkpl9" in resourcegroup="rg-np-dg-minor-8jkpl9-97vgtw4swh8v" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1dc855f6-8924-4660-af02-dc66ad12ebdd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1dc855f6-8924-4660-af02-dc66ad12ebdd", "name": "1dc855f6-8924-4660-af02-dc66ad12ebdd", "status": "Failed", "startTime": "2026-09-22T13:32:27.973621119Z", "endTime": "2026-09-22T14:01:34.946432956Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000bb8c80>: failed to create HCP cluster np-dg-minor-8jkpl9: failed waiting for cluster="np-dg-minor-8jkpl9" in resourcegroup="rg-np-dg-minor-8jkpl9-97vgtw4swh8v" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1dc855f6-8924-4660-af02-dc66ad12ebdd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1dc855f6-8924-4660-af02-dc66ad12ebdd", "name": "1dc855f6-8924-4660-af02-dc66ad12ebdd", "status": "Failed", "startTime": "2026-09-22T13:32:27.973621119Z", "endTime": "2026-09-22T14:01:34.946432956Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-xbw9gf Unexpected error: <fmt.wrapError \| 0xc000f22840>: failed to create HCP cluster np-noedge-xbw9gf: failed waiting for cluster="np-noedge-xbw9gf" in resourcegroup="rg-np-noedge-xbw9gf-2g5w2t9xwwpv" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3013569-9710-4d56-9127-660e8d0e8b4e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3013569-9710-4d56-9127-660e8d0e8b4e", "name": "e3013569-9710-4d56-9127-660e8d0e8b4e", "status": "Failed", "startTime": "2026-09-22T13:32:47.584413552Z", "endTime": "2026-09-22T14:01:54.952468852Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-xbw9gf Unexpected error: <fmt.wrapError \| 0xc000f22840>: failed to create HCP cluster np-noedge-xbw9gf: failed waiting for cluster="np-noedge-xbw9gf" in resourcegroup="rg-np-noedge-xbw9gf-2g5w2t9xwwpv" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3013569-9710-4d56-9127-660e8d0e8b4e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e3013569-9710-4d56-9127-660e8d0e8b4e", "name": "e3013569-9710-4d56-9127-660e8d0e8b4e", "status": "Failed", "startTime": "2026-09-22T13:32:47.584413552Z", "endTime": "2026-09-22T14:01:54.952468852Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle: Cluster provisioning failed 12:46:33 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-5jzdhm" with version 5.0.0-rc.2 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0010a2600>: failed to create HCP cluster cluster-zstream-5-0-5jzdhm: failed waiting for cluster="cluster-zstream-5-0-5jzdhm" in resourcegroup="rg-zstream-upgrade-5-0-kt8kw5kcc6ns" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/21ebda2b-c622-4b56-b92a-871dda0eed85 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/21ebda2b-c622-4b56-b92a-871dda0eed85", "name": "21ebda2b-c622-4b56-b92a-871dda0eed85", "status": "Failed", "startTime": "2026-09-22T13:32:01.665343445Z", "endTime": "2026-09-22T14:01:04.637676887Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-5jzdhm" with version 5.0.0-rc.2 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0010a2600>: failed to create HCP cluster cluster-zstream-5-0-5jzdhm: failed waiting for cluster="cluster-zstream-5-0-5jzdhm" in resourcegroup="rg-zstream-upgrade-5-0-kt8kw5kcc6ns" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/21ebda2b-c622-4b56-b92a-871dda0eed85 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/21ebda2b-c622-4b56-b92a-871dda0eed85", "name": "21ebda2b-c622-4b56-b92a-871dda0eed85", "status": "Failed", "startTime": "2026-09-22T13:32:01.665343445Z", "endTime": "2026-09-22T14:01:04.637676887Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000d87020>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-64pg9485vnxx" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d2fbe0d2-aad1-4d01-bb33-d19f8d8b8c62 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d2fbe0d2-aad1-4d01-bb33-d19f8d8b8c62", "name": "d2fbe0d2-aad1-4d01-bb33-d19f8d8b8c62", "status": "Failed", "startTime": "2026-09-22T13:32:28.884296464Z", "endTime": "2026-09-22T14:01:34.90419391Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000d87020>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-64pg9485vnxx" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d2fbe0d2-aad1-4d01-bb33-d19f8d8b8c62 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d2fbe0d2-aad1-4d01-bb33-d19f8d8b8c62", "name": "d2fbe0d2-aad1-4d01-bb33-d19f8d8b8c62", "status": "Failed", "startTime": "2026-09-22T13:32:28.884296464Z", "endTime": "2026-09-22T14:01:34.90419391Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:83]: failed to create HCP cluster np-update-nodes-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000de6840>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-htnbsdgmvcr4" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb215c1d-0a1a-47ba-a73d-78e5f47e89af -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb215c1d-0a1a-47ba-a73d-78e5f47e89af", "name": "cb215c1d-0a1a-47ba-a73d-78e5f47e89af", "status": "Failed", "startTime": "2026-09-22T13:32:27.21502087Z", "endTime": "2026-09-22T14:01:34.964236451Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:83]: failed to create HCP cluster np-update-nodes-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000de6840>: failed to create HCP cluster np-update-nodes-hcp-cluster: failed waiting for cluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-htnbsdgmvcr4" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb215c1d-0a1a-47ba-a73d-78e5f47e89af -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb215c1d-0a1a-47ba-a73d-78e5f47e89af", "name": "cb215c1d-0a1a-47ba-a73d-78e5f47e89af", "status": "Failed", "startTime": "2026-09-22T13:32:27.21502087Z", "endTime": "2026-09-22T14:01:34.964236451Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-mtqdfc with version 4.21.34 Unexpected error: <fmt.wrapError \| 0xc001a98480>: failed to create HCP cluster np-version-upgrade-cluster-mtqdfc: failed waiting for cluster="np-version-upgrade-cluster-mtqdfc" in resourcegroup="rg-np-version-upgrade-mtqdfc-bth59zhjnz6n" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f65ee424-6cf6-4989-a7d1-891071500c60 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f65ee424-6cf6-4989-a7d1-891071500c60", "name": "f65ee424-6cf6-4989-a7d1-891071500c60", "status": "Failed", "startTime": "2026-09-22T13:32:28.003057597Z", "endTime": "2026-09-22T14:01:34.644902606Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-mtqdfc with version 4.21.34 Unexpected error: <fmt.wrapError \| 0xc001a98480>: failed to create HCP cluster np-version-upgrade-cluster-mtqdfc: failed waiting for cluster="np-version-upgrade-cluster-mtqdfc" in resourcegroup="rg-np-version-upgrade-mtqdfc-bth59zhjnz6n" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f65ee424-6cf6-4989-a7d1-891071500c60 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f65ee424-6cf6-4989-a7d1-891071500c60", "name": "f65ee424-6cf6-4989-a7d1-891071500c60", "status": "Failed", "startTime": "2026-09-22T13:32:28.003057597Z", "endTime": "2026-09-22T14:01:34.644902606Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001606840>: failed to create HCP cluster np-downgrade-m7w6pj: failed waiting for cluster="np-downgrade-m7w6pj" in resourcegroup="rg-np-downgrade-m7w6pj-b4q6lx8fd9gs" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f56cada-f16e-4e2f-ac20-e517ffdaf734 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f56cada-f16e-4e2f-ac20-e517ffdaf734", "name": "8f56cada-f16e-4e2f-ac20-e517ffdaf734", "status": "Failed", "startTime": "2026-09-22T13:32:49.748056499Z", "endTime": "2026-09-22T14:01:54.973155468Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001606840>: failed to create HCP cluster np-downgrade-m7w6pj: failed waiting for cluster="np-downgrade-m7w6pj" in resourcegroup="rg-np-downgrade-m7w6pj-b4q6lx8fd9gs" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f56cada-f16e-4e2f-ac20-e517ffdaf734 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8f56cada-f16e-4e2f-ac20-e517ffdaf734", "name": "8f56cada-f16e-4e2f-ac20-e517ffdaf734", "status": "Failed", "startTime": "2026-09-22T13:32:49.748056499Z", "endTime": "2026-09-22T14:01:54.973155468Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000d58a20>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-g7q8vnwmjvvk" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ea0270f1-32ca-4b4d-a131-126d53d2800a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ea0270f1-32ca-4b4d-a131-126d53d2800a", "name": "ea0270f1-32ca-4b4d-a131-126d53d2800a", "status": "Failed", "startTime": "2026-09-22T14:06:37.339670093Z", "endTime": "2026-09-22T14:35:38.595765601Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000d58a20>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-g7q8vnwmjvvk" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ea0270f1-32ca-4b4d-a131-126d53d2800a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ea0270f1-32ca-4b4d-a131-126d53d2800a", "name": "ea0270f1-32ca-4b4d-a131-126d53d2800a", "status": "Failed", "startTime": "2026-09-22T14:06:37.339670093Z", "endTime": "2026-09-22T14:35:38.595765601Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-m6rtch with version 4.20.39 Unexpected error: <fmt.wrapError \| 0xc000a6e4a0>: failed to create HCP cluster np-version-upgrade-cluster-m6rtch: failed waiting for cluster="np-version-upgrade-cluster-m6rtch" in resourcegroup="rg-np-version-upgrade-m6rtch-kp76f866ck8n" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ea5323bc-7d7a-4cd2-9069-577a587fdb12 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ea5323bc-7d7a-4cd2-9069-577a587fdb12", "name": "ea5323bc-7d7a-4cd2-9069-577a587fdb12", "status": "Failed", "startTime": "2026-09-22T13:32:57.760074103Z", "endTime": "2026-09-22T14:02:04.634127705Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-m6rtch with version 4.20.39 Unexpected error: <fmt.wrapError \| 0xc000a6e4a0>: failed to create HCP cluster np-version-upgrade-cluster-m6rtch: failed waiting for cluster="np-version-upgrade-cluster-m6rtch" in resourcegroup="rg-np-version-upgrade-m6rtch-kp76f866ck8n" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ea5323bc-7d7a-4cd2-9069-577a587fdb12 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ea5323bc-7d7a-4cd2-9069-577a587fdb12", "name": "ea5323bc-7d7a-4cd2-9069-577a587fdb12", "status": "Failed", "startTime": "2026-09-22T13:32:57.760074103Z", "endTime": "2026-09-22T14:02:04.634127705Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000c90ba0>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9pjbfw8psrpq" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb5b03af-09d4-44d3-b02e-ce58d25b6b3b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb5b03af-09d4-44d3-b02e-ce58d25b6b3b", "name": "cb5b03af-09d4-44d3-b02e-ce58d25b6b3b", "status": "Failed", "startTime": "2026-09-22T13:32:01.886561261Z", "endTime": "2026-09-22T14:01:04.632033687Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000c90ba0>: failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9pjbfw8psrpq" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb5b03af-09d4-44d3-b02e-ce58d25b6b3b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cb5b03af-09d4-44d3-b02e-ce58d25b6b3b", "name": "cb5b03af-09d4-44d3-b02e-ce58d25b6b3b", "status": "Failed", "startTime": "2026-09-22T13:32:01.886561261Z", "endTime": "2026-09-22T14:01:04.632033687Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-th49nv with version 4.21.34 Unexpected error: <fmt.wrapError \| 0xc000def320>: failed to create HCP cluster np-version-upgrade-cluster-th49nv: failed waiting for cluster="np-version-upgrade-cluster-th49nv" in resourcegroup="rg-np-version-upgrade-th49nv-j4l6x9jgvj9t" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/80f13221-7309-42c2-8783-0028aa3526c6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/80f13221-7309-42c2-8783-0028aa3526c6", "name": "80f13221-7309-42c2-8783-0028aa3526c6", "status": "Failed", "startTime": "2026-09-22T13:32:48.394279998Z", "endTime": "2026-09-22T14:01:54.565780738Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-th49nv with version 4.21.34 Unexpected error: <fmt.wrapError \| 0xc000def320>: failed to create HCP cluster np-version-upgrade-cluster-th49nv: failed waiting for cluster="np-version-upgrade-cluster-th49nv" in resourcegroup="rg-np-version-upgrade-th49nv-j4l6x9jgvj9t" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/80f13221-7309-42c2-8783-0028aa3526c6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/80f13221-7309-42c2-8783-0028aa3526c6", "name": "80f13221-7309-42c2-8783-0028aa3526c6", "status": "Failed", "startTime": "2026-09-22T13:32:48.394279998Z", "endTime": "2026-09-22T14:01:54.565780738Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0xc000958220>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-92jh9cwk45qj" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20b25f8b-196a-49c8-a2f2-8e0d2717aa00 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20b25f8b-196a-49c8-a2f2-8e0d2717aa00", "name": "20b25f8b-196a-49c8-a2f2-8e0d2717aa00", "status": "Failed", "startTime": "2026-09-22T13:33:15.551368137Z", "endTime": "2026-09-22T14:02:24.66646054Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0xc000958220>: failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-92jh9cwk45qj" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20b25f8b-196a-49c8-a2f2-8e0d2717aa00 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/20b25f8b-196a-49c8-a2f2-8e0d2717aa00", "name": "20b25f8b-196a-49c8-a2f2-8e0d2717aa00", "status": "Failed", "startTime": "2026-09-22T13:33:15.551368137Z", "endTime": "2026-09-22T14:02:24.66646054Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-rmbgws" at install version 4.20 Unexpected error: <fmt.wrapError \| 0xc001702980>: failed to create HCP cluster cp-ystream-upgrade-4-21-rmbgws: failed waiting for cluster="cp-ystream-upgrade-4-21-rmbgws" in resourcegroup="rg-cp-ystream-upgrade-4-21-z5xs8zlcnjkk" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f6e84d1d-17fd-4447-b870-5ab61c77bdf1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f6e84d1d-17fd-4447-b870-5ab61c77bdf1", "name": "f6e84d1d-17fd-4447-b870-5ab61c77bdf1", "status": "Failed", "startTime": "2026-09-22T13:32:19.46291759Z", "endTime": "2026-09-22T14:01:24.769248375Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-rmbgws" at install version 4.20 Unexpected error: <fmt.wrapError \| 0xc001702980>: failed to create HCP cluster cp-ystream-upgrade-4-21-rmbgws: failed waiting for cluster="cp-ystream-upgrade-4-21-rmbgws" in resourcegroup="rg-cp-ystream-upgrade-4-21-z5xs8zlcnjkk" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f6e84d1d-17fd-4447-b870-5ab61c77bdf1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f6e84d1d-17fd-4447-b870-5ab61c77bdf1", "name": "f6e84d1d-17fd-4447-b870-5ab61c77bdf1", "status": "Failed", "startTime": "2026-09-22T13:32:19.46291759Z", "endTime": "2026-09-22T14:01:24.769248375Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000ce8920>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-4th9vj6whvx7" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30794bc9-050b-4975-a1bb-7ea957f163fb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30794bc9-050b-4975-a1bb-7ea957f163fb", "name": "30794bc9-050b-4975-a1bb-7ea957f163fb", "status": "Failed", "startTime": "2026-09-22T13:32:20.953844818Z", "endTime": "2026-09-22T14:01:24.858958675Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000ce8920>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-4th9vj6whvx7" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30794bc9-050b-4975-a1bb-7ea957f163fb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/30794bc9-050b-4975-a1bb-7ea957f163fb", "name": "30794bc9-050b-4975-a1bb-7ea957f163fb", "status": "Failed", "startTime": "2026-09-22T13:32:20.953844818Z", "endTime": "2026-09-22T14:01:24.858958675Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc00148a820>: failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-ql6nc95drl8l" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7d253a4-df32-4561-8861-2e0882591562 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7d253a4-df32-4561-8861-2e0882591562", "name": "f7d253a4-df32-4561-8861-2e0882591562", "status": "Failed", "startTime": "2026-09-22T13:32:48.802075858Z", "endTime": "2026-09-22T14:01:54.971192416Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc00148a820>: failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-ql6nc95drl8l" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7d253a4-df32-4561-8861-2e0882591562 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f7d253a4-df32-4561-8861-2e0882591562", "name": "f7d253a4-df32-4561-8861-2e0882591562", "status": "Failed", "startTime": "2026-09-22T13:32:48.802075858Z", "endTime": "2026-09-22T14:01:54.971192416Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 12:46:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20 Unexpected error: <fmt.wrapErrors \| 0xc000a123f0>: failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-57bkrhmpnrrx" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-57bkrhmpnrrx, error: context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20 Unexpected error: <fmt.wrapErrors \| 0xc000a123f0>: failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-57bkrhmpnrrx" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-57bkrhmpnrrx, error: context deadline exceeded ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--n5xwmlbc98xb/cluster-hs-pre-b9ffdp should provision Unexpected error: <fmt.wrapError \| 0xc00108eaa0>: failed to create HCP cluster cluster-hs-pre-b9ffdp: failed waiting for cluster="cluster-hs-pre-b9ffdp" in resourcegroup="rg-hs-pre--n5xwmlbc98xb" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e1c451f0-78db-458f-9923-7716b40c6c90 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e1c451f0-78db-458f-9923-7716b40c6c90", "name": "e1c451f0-78db-458f-9923-7716b40c6c90", "status": "Failed", "startTime": "2026-09-22T13:32:22.063711473Z", "endTime": "2026-09-22T14:01:24.664579405Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--n5xwmlbc98xb/cluster-hs-pre-b9ffdp should provision Unexpected error: <fmt.wrapError \| 0xc00108eaa0>: failed to create HCP cluster cluster-hs-pre-b9ffdp: failed waiting for cluster="cluster-hs-pre-b9ffdp" in resourcegroup="rg-hs-pre--n5xwmlbc98xb" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e1c451f0-78db-458f-9923-7716b40c6c90 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e1c451f0-78db-458f-9923-7716b40c6c90", "name": "e1c451f0-78db-458f-9923-7716b40c6c90", "status": "Failed", "startTime": "2026-09-22T13:32:22.063711473Z", "endTime": "2026-09-22T14:01:24.664579405Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test SRE should be able to retrieve serial console logs for a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test Unexpected error: <fmt.wrapError \| 0xc00108a1c0>: failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-2tnmgccpx9km" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bfe5c5f3-e516-408f-8ab3-b1c6d692820b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bfe5c5f3-e516-408f-8ab3-b1c6d692820b", "name": "bfe5c5f3-e516-408f-8ab3-b1c6d692820b", "status": "Failed", "startTime": "2026-09-22T13:32:22.208739639Z", "endTime": "2026-09-22T14:01:24.939813833Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test Unexpected error: <fmt.wrapError \| 0xc00108a1c0>: failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-2tnmgccpx9km" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bfe5c5f3-e516-408f-8ab3-b1c6d692820b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bfe5c5f3-e516-408f-8ab3-b1c6d692820b", "name": "bfe5c5f3-e516-408f-8ab3-b1c6d692820b", "status": "Failed", "startTime": "2026-09-22T13:32:22.208739639Z", "endTime": "2026-09-22T14:01:24.939813833Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000a08be0>: failed to create HCP cluster ea-listcqmtxr: failed waiting for cluster="ea-listcqmtxr" in resourcegroup="ea-list-542l4vff7xkq" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef207e33-9e75-44d1-9a69-d8b4514144d0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef207e33-9e75-44d1-9a69-d8b4514144d0", "name": "ef207e33-9e75-44d1-9a69-d8b4514144d0", "status": "Failed", "startTime": "2026-09-22T13:32:25.585548746Z", "endTime": "2026-09-22T14:01:34.938634584Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000a08be0>: failed to create HCP cluster ea-listcqmtxr: failed waiting for cluster="ea-listcqmtxr" in resourcegroup="ea-list-542l4vff7xkq" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef207e33-9e75-44d1-9a69-d8b4514144d0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ef207e33-9e75-44d1-9a69-d8b4514144d0", "name": "ef207e33-9e75-44d1-9a69-d8b4514144d0", "status": "Failed", "startTime": "2026-09-22T13:32:25.585548746Z", "endTime": "2026-09-22T14:01:34.938634584Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc001250920>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-6jbj2h648xbl" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c35ce14b-2ef3-4651-bdad-b2703112d6f1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c35ce14b-2ef3-4651-bdad-b2703112d6f1", "name": "c35ce14b-2ef3-4651-bdad-b2703112d6f1", "status": "Failed", "startTime": "2026-09-22T13:34:52.214220949Z", "endTime": "2026-09-22T14:04:02.008733809Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc001250920>: failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-6jbj2h648xbl" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c35ce14b-2ef3-4651-bdad-b2703112d6f1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c35ce14b-2ef3-4651-bdad-b2703112d6f1", "name": "c35ce14b-2ef3-4651-bdad-b2703112d6f1", "status": "Failed", "startTime": "2026-09-22T13:34:52.214220949Z", "endTime": "2026-09-22T14:04:02.008733809Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-sxjznjq8jpsn/cluster-candidate-4-22-wmljjz should provision Unexpected error: <fmt.wrapError \| 0xc001122120>: failed to create HCP cluster cluster-candidate-4-22-wmljjz: failed waiting for cluster="cluster-candidate-4-22-wmljjz" in resourcegroup="rg-candidate-4-22-sxjznjq8jpsn" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7addf21d-a67b-4865-b5b3-afec4261e5ef -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7addf21d-a67b-4865-b5b3-afec4261e5ef", "name": "7addf21d-a67b-4865-b5b3-afec4261e5ef", "status": "Failed", "startTime": "2026-09-22T13:32:21.967597365Z", "endTime": "2026-09-22T14:01:24.832286014Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-sxjznjq8jpsn/cluster-candidate-4-22-wmljjz should provision Unexpected error: <fmt.wrapError \| 0xc001122120>: failed to create HCP cluster cluster-candidate-4-22-wmljjz: failed waiting for cluster="cluster-candidate-4-22-wmljjz" in resourcegroup="rg-candidate-4-22-sxjznjq8jpsn" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7addf21d-a67b-4865-b5b3-afec4261e5ef -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7addf21d-a67b-4865-b5b3-afec4261e5ef", "name": "7addf21d-a67b-4865-b5b3-afec4261e5ef", "status": "Failed", "startTime": "2026-09-22T13:32:21.967597365Z", "endTime": "2026-09-22T14:01:24.832286014Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc0009f8660>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-wnpc7fdg4jjt" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95fff587-94f2-447e-bc0d-a6f5af7dd01f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95fff587-94f2-447e-bc0d-a6f5af7dd01f", "name": "95fff587-94f2-447e-bc0d-a6f5af7dd01f", "status": "Failed", "startTime": "2026-09-22T13:32:57.870353747Z", "endTime": "2026-09-22T14:02:04.600472123Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc0009f8660>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-wnpc7fdg4jjt" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95fff587-94f2-447e-bc0d-a6f5af7dd01f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95fff587-94f2-447e-bc0d-a6f5af7dd01f", "name": "95fff587-94f2-447e-bc0d-a6f5af7dd01f", "status": "Failed", "startTime": "2026-09-22T13:32:57.870353747Z", "endTime": "2026-09-22T14:02:04.600472123Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000c5ecc0>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-q8wnpjlbcjd2" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77d0c1ba-ef9b-44da-bac1-09a4d6bc9c79 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77d0c1ba-ef9b-44da-bac1-09a4d6bc9c79", "name": "77d0c1ba-ef9b-44da-bac1-09a4d6bc9c79", "status": "Failed", "startTime": "2026-09-22T13:32:20.803439313Z", "endTime": "2026-09-22T14:01:24.833690805Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000c5ecc0>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-q8wnpjlbcjd2" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77d0c1ba-ef9b-44da-bac1-09a4d6bc9c79 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/77d0c1ba-ef9b-44da-bac1-09a4d6bc9c79", "name": "77d0c1ba-ef9b-44da-bac1-09a4d6bc9c79", "status": "Failed", "startTime": "2026-09-22T13:32:20.803439313Z", "endTime": "2026-09-22T14:01:24.833690805Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to retrieve cluster and nodepool status active versions · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:88]: failed to create HCP cluster "v20261001" via v20261001preview Unexpected error: <fmt.wrapError \| 0xc000b77d80>: failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-79cwkrjwz9cs" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba01c81e-c5a2-45f4-bd07-7eb9f1de61bf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba01c81e-c5a2-45f4-bd07-7eb9f1de61bf", "name": "ba01c81e-c5a2-45f4-bd07-7eb9f1de61bf", "status": "Failed", "startTime": "2026-09-22T13:32:22.58553866Z", "endTime": "2026-09-22T14:01:24.895864789Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:88]: failed to create HCP cluster "v20261001" via v20261001preview Unexpected error: <fmt.wrapError \| 0xc000b77d80>: failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-79cwkrjwz9cs" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba01c81e-c5a2-45f4-bd07-7eb9f1de61bf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba01c81e-c5a2-45f4-bd07-7eb9f1de61bf", "name": "ba01c81e-c5a2-45f4-bd07-7eb9f1de61bf", "status": "Failed", "startTime": "2026-09-22T13:32:22.58553866Z", "endTime": "2026-09-22T14:01:24.895864789Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-7sr4g9qqtq49/cluster-candidate-4-21-n4s875 should provision Unexpected error: <fmt.wrapError \| 0xc000f7a4e0>: failed to create HCP cluster cluster-candidate-4-21-n4s875: failed waiting for cluster="cluster-candidate-4-21-n4s875" in resourcegroup="rg-candidate-4-21-7sr4g9qqtq49" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/41bada58-020b-4e77-89a8-255bde10db94 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/41bada58-020b-4e77-89a8-255bde10db94", "name": "41bada58-020b-4e77-89a8-255bde10db94", "status": "Failed", "startTime": "2026-09-22T13:31:50.90986267Z", "endTime": "2026-09-22T14:00:54.82147705Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-7sr4g9qqtq49/cluster-candidate-4-21-n4s875 should provision Unexpected error: <fmt.wrapError \| 0xc000f7a4e0>: failed to create HCP cluster cluster-candidate-4-21-n4s875: failed waiting for cluster="cluster-candidate-4-21-n4s875" in resourcegroup="rg-candidate-4-21-7sr4g9qqtq49" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/41bada58-020b-4e77-89a8-255bde10db94 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/41bada58-020b-4e77-89a8-255bde10db94", "name": "41bada58-020b-4e77-89a8-255bde10db94", "status": "Failed", "startTime": "2026-09-22T13:31:50.90986267Z", "endTime": "2026-09-22T14:00:54.82147705Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc00102e520>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-ccvxf6h9dzgs" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ee709e4b-d97c-4079-933f-9b1ba906ba99 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ee709e4b-d97c-4079-933f-9b1ba906ba99", "name": "ee709e4b-d97c-4079-933f-9b1ba906ba99", "status": "Failed", "startTime": "2026-09-22T13:32:46.874292726Z", "endTime": "2026-09-22T14:01:54.873323023Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc00102e520>: failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-ccvxf6h9dzgs" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ee709e4b-d97c-4079-933f-9b1ba906ba99 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ee709e4b-d97c-4079-933f-9b1ba906ba99", "name": "ee709e4b-d97c-4079-933f-9b1ba906ba99", "status": "Failed", "startTime": "2026-09-22T13:32:46.874292726Z", "endTime": "2026-09-22T14:01:54.873323023Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000fea300>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-s6f9ps8n479f" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd8729f7-dafd-47d9-b4c6-5199ca0e7104 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd8729f7-dafd-47d9-b4c6-5199ca0e7104", "name": "cd8729f7-dafd-47d9-b4c6-5199ca0e7104", "status": "Failed", "startTime": "2026-09-22T13:32:47.611769362Z", "endTime": "2026-09-22T14:01:54.819460445Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc000fea300>: failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-s6f9ps8n479f" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd8729f7-dafd-47d9-b4c6-5199ca0e7104 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cd8729f7-dafd-47d9-b4c6-5199ca0e7104", "name": "cd8729f7-dafd-47d9-b4c6-5199ca0e7104", "status": "Failed", "startTime": "2026-09-22T13:32:47.611769362Z", "endTime": "2026-09-22T14:01:54.819460445Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc001022b80>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-fmbd5h9tdkzn" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a185f7e-c196-4cca-9649-3a45bca25626 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a185f7e-c196-4cca-9649-3a45bca25626", "name": "6a185f7e-c196-4cca-9649-3a45bca25626", "status": "Failed", "startTime": "2026-09-22T13:32:01.53469242Z", "endTime": "2026-09-22T14:01:04.986880418Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc001022b80>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-fmbd5h9tdkzn" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a185f7e-c196-4cca-9649-3a45bca25626 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6a185f7e-c196-4cca-9649-3a45bca25626", "name": "6a185f7e-c196-4cca-9649-3a45bca25626", "status": "Failed", "startTime": "2026-09-22T13:32:01.53469242Z", "endTime": "2026-09-22T14:01:04.986880418Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc00072e440>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-lmh8xtrqwsz8" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e94fb788-b80f-4ab9-98a4-a704b55c9089 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e94fb788-b80f-4ab9-98a4-a704b55c9089", "name": "e94fb788-b80f-4ab9-98a4-a704b55c9089", "status": "Failed", "startTime": "2026-09-22T13:32:35.011447868Z", "endTime": "2026-09-22T14:01:44.849684341Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test Unexpected error: <fmt.wrapError \| 0xc00072e440>: failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-lmh8xtrqwsz8" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e94fb788-b80f-4ab9-98a4-a704b55c9089 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e94fb788-b80f-4ab9-98a4-a704b55c9089", "name": "e94fb788-b80f-4ab9-98a4-a704b55c9089", "status": "Failed", "startTime": "2026-09-22T13:32:35.011447868Z", "endTime": "2026-09-22T14:01:44.849684341Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-2cjqpskxzwwh/cluster-candidate-4-20-gdcb55 should provision Unexpected error: <fmt.wrapError \| 0xc00117c120>: failed to create HCP cluster cluster-candidate-4-20-gdcb55: failed waiting for cluster="cluster-candidate-4-20-gdcb55" in resourcegroup="rg-candidate-4-20-2cjqpskxzwwh" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/403eb990-4d95-45bc-b1d8-871f6a3f66f3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/403eb990-4d95-45bc-b1d8-871f6a3f66f3", "name": "403eb990-4d95-45bc-b1d8-871f6a3f66f3", "status": "Failed", "startTime": "2026-09-22T13:32:22.904006297Z", "endTime": "2026-09-22T14:01:24.628150051Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-2cjqpskxzwwh/cluster-candidate-4-20-gdcb55 should provision Unexpected error: <fmt.wrapError \| 0xc00117c120>: failed to create HCP cluster cluster-candidate-4-20-gdcb55: failed waiting for cluster="cluster-candidate-4-20-gdcb55" in resourcegroup="rg-candidate-4-20-2cjqpskxzwwh" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/403eb990-4d95-45bc-b1d8-871f6a3f66f3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/403eb990-4d95-45bc-b1d8-871f6a3f66f3", "name": "403eb990-4d95-45bc-b1d8-871f6a3f66f3", "status": "Failed", "startTime": "2026-09-22T13:32:22.904006297Z", "endTime": "2026-09-22T14:01:24.628150051Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test SRE can pause schedules to stop backup execution for an HCP cluster · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000ce8a00>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-7b2cjlpnvnvc" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/48483c11-8284-4a72-b0e8-808c25d5a875 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/48483c11-8284-4a72-b0e8-808c25d5a875", "name": "48483c11-8284-4a72-b0e8-808c25d5a875", "status": "Failed", "startTime": "2026-09-22T13:32:27.071336756Z", "endTime": "2026-09-22T14:01:34.588882658Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000ce8a00>: failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-7b2cjlpnvnvc" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/48483c11-8284-4a72-b0e8-808c25d5a875 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/48483c11-8284-4a72-b0e8-808c25d5a875", "name": "48483c11-8284-4a72-b0e8-808c25d5a875", "status": "Failed", "startTime": "2026-09-22T13:32:27.071336756Z", "endTime": "2026-09-22T14:01:34.588882658Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0010c0b80>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-qwx5tf2dtpk2" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e31a2de-3b83-48d4-a41a-2116a2e676ce -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e31a2de-3b83-48d4-a41a-2116a2e676ce", "name": "5e31a2de-3b83-48d4-a41a-2116a2e676ce", "status": "Failed", "startTime": "2026-09-22T13:32:01.635132061Z", "endTime": "2026-09-22T14:01:04.903065926Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc0010c0b80>: failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-qwx5tf2dtpk2" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e31a2de-3b83-48d4-a41a-2116a2e676ce -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5e31a2de-3b83-48d4-a41a-2116a2e676ce", "name": "5e31a2de-3b83-48d4-a41a-2116a2e676ce", "status": "Failed", "startTime": "2026-09-22T13:32:01.635132061Z", "endTime": "2026-09-22T14:01:04.903065926Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test SRE should return 409 when boot diagnostics is disabled on a VM · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test Unexpected error: <fmt.wrapError \| 0xc001656760>: failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-m9xd8z5rbsjl" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276a3d3f-6f71-48de-a173-d167c5d98fe6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276a3d3f-6f71-48de-a173-d167c5d98fe6", "name": "276a3d3f-6f71-48de-a173-d167c5d98fe6", "status": "Failed", "startTime": "2026-09-22T13:32:02.228959747Z", "endTime": "2026-09-22T14:01:04.56945549Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:417]: failed to create HCP cluster "sre-hcp-cluster-bootdiag" for boot diagnostics test Unexpected error: <fmt.wrapError \| 0xc001656760>: failed to create HCP cluster sre-hcp-cluster-bootdiag: failed waiting for cluster="sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-m9xd8z5rbsjl" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276a3d3f-6f71-48de-a173-d167c5d98fe6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/276a3d3f-6f71-48de-a173-d167c5d98fe6", "name": "276a3d3f-6f71-48de-a173-d167c5d98fe6", "status": "Failed", "startTime": "2026-09-22T13:32:02.228959747Z", "endTime": "2026-09-22T14:01:04.56945549Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Engineering should be able to retrieve kusto logs for a cluster and services · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:88]: failed to create HCP cluster for kusto logs test Unexpected error: <fmt.wrapError \| 0xc00039ed80>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-ln96hxc65p2r" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0a399061-f866-4bd2-911c-6ed39af5b54d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0a399061-f866-4bd2-911c-6ed39af5b54d", "name": "0a399061-f866-4bd2-911c-6ed39af5b54d", "status": "Failed", "startTime": "2026-09-22T13:32:17.617030211Z", "endTime": "2026-09-22T14:01:24.711758901Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/kusto_logs_present.go:88]: failed to create HCP cluster for kusto logs test Unexpected error: <fmt.wrapError \| 0xc00039ed80>: failed to create HCP cluster engineering-hcp-cluster: failed waiting for cluster="engineering-hcp-cluster" in resourcegroup="kusto-logs-ln96hxc65p2r" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0a399061-f866-4bd2-911c-6ed39af5b54d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0a399061-f866-4bd2-911c-6ed39af5b54d", "name": "0a399061-f866-4bd2-911c-6ed39af5b54d", "status": "Failed", "startTime": "2026-09-22T13:32:17.617030211Z", "endTime": "2026-09-22T14:01:24.711758901Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000eac380>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-gg78jpjwb6l4" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/27a48f2f-dae8-4b01-9ddd-35f85f532068 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/27a48f2f-dae8-4b01-9ddd-35f85f532068", "name": "27a48f2f-dae8-4b01-9ddd-35f85f532068", "status": "Failed", "startTime": "2026-09-22T13:32:38.178380757Z", "endTime": "2026-09-22T14:01:44.75918192Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000eac380>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-gg78jpjwb6l4" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/27a48f2f-dae8-4b01-9ddd-35f85f532068 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/27a48f2f-dae8-4b01-9ddd-35f85f532068", "name": "27a48f2f-dae8-4b01-9ddd-35f85f532068", "status": "Failed", "startTime": "2026-09-22T13:32:38.178380757Z", "endTime": "2026-09-22T14:01:44.75918192Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc000e4a200>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-cxwvpl2jz4z4" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7233baf0-2045-40c1-a77e-4bcc134eee10 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7233baf0-2045-40c1-a77e-4bcc134eee10", "name": "7233baf0-2045-40c1-a77e-4bcc134eee10", "status": "Failed", "startTime": "2026-09-22T13:32:22.052170362Z", "endTime": "2026-09-22T14:01:24.90059412Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc000e4a200>: failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-cxwvpl2jz4z4" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7233baf0-2045-40c1-a77e-4bcc134eee10 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7233baf0-2045-40c1-a77e-4bcc134eee10", "name": "7233baf0-2045-40c1-a77e-4bcc134eee10", "status": "Failed", "startTime": "2026-09-22T13:32:22.052170362Z", "endTime": "2026-09-22T14:01:24.90059412Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc0007c0de0>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-q4zkjf7prwjv" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a82356a5-5cf7-4422-8f08-898257e76006 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a82356a5-5cf7-4422-8f08-898257e76006", "name": "a82356a5-5cf7-4422-8f08-898257e76006", "status": "Failed", "startTime": "2026-09-22T13:32:17.380507861Z", "endTime": "2026-09-22T14:01:24.867942166Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc0007c0de0>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-q4zkjf7prwjv" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a82356a5-5cf7-4422-8f08-898257e76006 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a82356a5-5cf7-4422-8f08-898257e76006", "name": "a82356a5-5cf7-4422-8f08-898257e76006", "status": "Failed", "startTime": "2026-09-22T13:32:17.380507861Z", "endTime": "2026-09-22T14:01:24.867942166Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-rld9nl" with version 4.20.38 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000ca4560>: failed to create HCP cluster cluster-zstream-4-20-rld9nl: failed waiting for cluster="cluster-zstream-4-20-rld9nl" in resourcegroup="rg-zstream-upgrade-4-20-5pqqs67q87zt" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa4828c5-ed08-491a-9850-88300b44f617 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa4828c5-ed08-491a-9850-88300b44f617", "name": "aa4828c5-ed08-491a-9850-88300b44f617", "status": "Failed", "startTime": "2026-09-22T13:32:17.855427061Z", "endTime": "2026-09-22T14:01:24.824477819Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-rld9nl" with version 4.20.38 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000ca4560>: failed to create HCP cluster cluster-zstream-4-20-rld9nl: failed waiting for cluster="cluster-zstream-4-20-rld9nl" in resourcegroup="rg-zstream-upgrade-4-20-5pqqs67q87zt" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa4828c5-ed08-491a-9850-88300b44f617 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/aa4828c5-ed08-491a-9850-88300b44f617", "name": "aa4828c5-ed08-491a-9850-88300b44f617", "status": "Failed", "startTime": "2026-09-22T13:32:17.855427061Z", "endTime": "2026-09-22T14:01:24.824477819Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0xc000b6ba80>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-7nfnl9sxpnw4" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db771643-cb7b-4400-b0be-f0453c7d0b78 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db771643-cb7b-4400-b0be-f0453c7d0b78", "name": "db771643-cb7b-4400-b0be-f0453c7d0b78", "status": "Failed", "startTime": "2026-09-22T13:32:06.730860572Z", "endTime": "2026-09-22T14:01:14.792387469Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0xc000b6ba80>: failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-7nfnl9sxpnw4" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db771643-cb7b-4400-b0be-f0453c7d0b78 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/db771643-cb7b-4400-b0be-f0453c7d0b78", "name": "db771643-cb7b-4400-b0be-f0453c7d0b78", "status": "Failed", "startTime": "2026-09-22T13:32:06.730860572Z", "endTime": "2026-09-22T14:01:14.792387469Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc001242760>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-cqg4vcwtml8g" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b15f1682-d00f-480f-b189-4cfab6cb1f2d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b15f1682-d00f-480f-b189-4cfab6cb1f2d", "name": "b15f1682-d00f-480f-b189-4cfab6cb1f2d", "status": "Failed", "startTime": "2026-09-22T13:32:13.858451659Z", "endTime": "2026-09-22T14:01:14.879913195Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc001242760>: failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-cqg4vcwtml8g" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b15f1682-d00f-480f-b189-4cfab6cb1f2d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b15f1682-d00f-480f-b189-4cfab6cb1f2d", "name": "b15f1682-d00f-480f-b189-4cfab6cb1f2d", "status": "Failed", "startTime": "2026-09-22T13:32:13.858451659Z", "endTime": "2026-09-22T14:01:14.879913195Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred DataCurrent condition must be True 12:46:33 UTC · failed at e2e · test Fleet should have HCP resource requirements data · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:115]: Timed out after 1800.001s. HCP resource requirements were not populated in time The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:105 with: DataCurrent condition must be True Expected <bool>: ... to be true fail [github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:115]: Timed out after 1800.001s. HCP resource requirements were not populated in time The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/fleet_registration.go:105 with: DataCurrent condition must be True Expected <bool>: ... to be true ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0xc000924cc0>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-smvxtvkx87p2" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5181718b-c651-4b64-b71c-5dfd1c86ee96 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5181718b-c651-4b64-b71c-5dfd1c86ee96", "name": "5181718b-c651-4b64-b71c-5dfd1c86ee96", "status": "Failed", "startTime": "2026-09-22T13:32:53.578618144Z", "endTime": "2026-09-22T14:01:54.769879975Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0xc000924cc0>: failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-smvxtvkx87p2" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5181718b-c651-4b64-b71c-5dfd1c86ee96 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5181718b-c651-4b64-b71c-5dfd1c86ee96", "name": "5181718b-c651-4b64-b71c-5dfd1c86ee96", "status": "Failed", "startTime": "2026-09-22T13:32:53.578618144Z", "endTime": "2026-09-22T14:01:54.769879975Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-qwtrff" with version 4.22.13 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0011f6780>: failed to create HCP cluster cluster-zstream-4-22-qwtrff: failed waiting for cluster="cluster-zstream-4-22-qwtrff" in resourcegroup="rg-zstream-upgrade-4-22-9ntddsd5mwjb" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f691bca0-b85d-45cf-8de4-554ba3049b83 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f691bca0-b85d-45cf-8de4-554ba3049b83", "name": "f691bca0-b85d-45cf-8de4-554ba3049b83", "status": "Failed", "startTime": "2026-09-22T13:32:53.959725008Z", "endTime": "2026-09-22T14:01:54.970928051Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-qwtrff" with version 4.22.13 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0011f6780>: failed to create HCP cluster cluster-zstream-4-22-qwtrff: failed waiting for cluster="cluster-zstream-4-22-qwtrff" in resourcegroup="rg-zstream-upgrade-4-22-9ntddsd5mwjb" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f691bca0-b85d-45cf-8de4-554ba3049b83 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f691bca0-b85d-45cf-8de4-554ba3049b83", "name": "f691bca0-b85d-45cf-8de4-554ba3049b83", "status": "Failed", "startTime": "2026-09-22T13:32:53.959725008Z", "endTime": "2026-09-22T14:01:54.970928051Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc0016e8c40>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-rjq4p7796qpn" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67f6727d-867f-44c6-99f9-65659d41500e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67f6727d-867f-44c6-99f9-65659d41500e", "name": "67f6727d-867f-44c6-99f9-65659d41500e", "status": "Failed", "startTime": "2026-09-22T13:32:11.313848844Z", "endTime": "2026-09-22T14:01:14.727166871Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc0016e8c40>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-rjq4p7796qpn" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67f6727d-867f-44c6-99f9-65659d41500e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/67f6727d-867f-44c6-99f9-65659d41500e", "name": "67f6727d-867f-44c6-99f9-65659d41500e", "status": "Failed", "startTime": "2026-09-22T13:32:11.313848844Z", "endTime": "2026-09-22T14:01:14.727166871Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-qkvr8k" at install version 4.22 Unexpected error: <fmt.wrapError \| 0xc0009f85e0>: failed to create HCP cluster cp-ystream-upgrade-5-0-qkvr8k: failed waiting for cluster="cp-ystream-upgrade-5-0-qkvr8k" in resourcegroup="rg-cp-ystream-upgrade-5-0-4n8dcjk4zbh7" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cff55541-6190-4c33-afc8-f052dfe5eacf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cff55541-6190-4c33-afc8-f052dfe5eacf", "name": "cff55541-6190-4c33-afc8-f052dfe5eacf", "status": "Failed", "startTime": "2026-09-22T13:32:11.583499194Z", "endTime": "2026-09-22T14:01:14.703665149Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-qkvr8k" at install version 4.22 Unexpected error: <fmt.wrapError \| 0xc0009f85e0>: failed to create HCP cluster cp-ystream-upgrade-5-0-qkvr8k: failed waiting for cluster="cp-ystream-upgrade-5-0-qkvr8k" in resourcegroup="rg-cp-ystream-upgrade-5-0-4n8dcjk4zbh7" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cff55541-6190-4c33-afc8-f052dfe5eacf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/cff55541-6190-4c33-afc8-f052dfe5eacf", "name": "cff55541-6190-4c33-afc8-f052dfe5eacf", "status": "Failed", "startTime": "2026-09-22T13:32:11.583499194Z", "endTime": "2026-09-22T14:01:14.703665149Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc000a72f40>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-rvlj9pd4g49j" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c57dc13f-c997-4327-af90-f5f1ab3c0f39 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c57dc13f-c997-4327-af90-f5f1ab3c0f39", "name": "c57dc13f-c997-4327-af90-f5f1ab3c0f39", "status": "Failed", "startTime": "2026-09-22T13:32:58.722624927Z", "endTime": "2026-09-22T14:02:04.725282332Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc000a72f40>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-rvlj9pd4g49j" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c57dc13f-c997-4327-af90-f5f1ab3c0f39 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c57dc13f-c997-4327-af90-f5f1ab3c0f39", "name": "c57dc13f-c997-4327-af90-f5f1ab3c0f39", "status": "Failed", "startTime": "2026-09-22T13:32:58.722624927Z", "endTime": "2026-09-22T14:02:04.725282332Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-29hg8c at version 4.22.14 Unexpected error: <fmt.wrapError \| 0xc000d92c00>: failed to create HCP cluster np-skip-minor-29hg8c: failed waiting for cluster="np-skip-minor-29hg8c" in resourcegroup="rg-np-skip-minor-29hg8c-4dgpbcj69fqv" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/53fab93e-9091-4e72-a485-8d1c0068d6fb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/53fab93e-9091-4e72-a485-8d1c0068d6fb", "name": "53fab93e-9091-4e72-a485-8d1c0068d6fb", "status": "Failed", "startTime": "2026-09-22T13:33:08.629407608Z", "endTime": "2026-09-22T14:02:14.714971948Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-29hg8c at version 4.22.14 Unexpected error: <fmt.wrapError \| 0xc000d92c00>: failed to create HCP cluster np-skip-minor-29hg8c: failed waiting for cluster="np-skip-minor-29hg8c" in resourcegroup="rg-np-skip-minor-29hg8c-4dgpbcj69fqv" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/53fab93e-9091-4e72-a485-8d1c0068d6fb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/53fab93e-9091-4e72-a485-8d1c0068d6fb", "name": "53fab93e-9091-4e72-a485-8d1c0068d6fb", "status": "Failed", "startTime": "2026-09-22T13:33:08.629407608Z", "endTime": "2026-09-22T14:02:14.714971948Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-krw7bcv4h9tp/cluster-candidate-5-0-npd8xs should provision Unexpected error: <fmt.wrapError \| 0xc00104eb20>: failed to create HCP cluster cluster-candidate-5-0-npd8xs: failed waiting for cluster="cluster-candidate-5-0-npd8xs" in resourcegroup="rg-candidate-5-0-krw7bcv4h9tp" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/677c6d18-0a88-45ea-8eef-dcf3bcaf9526 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/677c6d18-0a88-45ea-8eef-dcf3bcaf9526", "name": "677c6d18-0a88-45ea-8eef-dcf3bcaf9526", "status": "Failed", "startTime": "2026-09-22T13:32:01.537906113Z", "endTime": "2026-09-22T14:01:04.991194141Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-krw7bcv4h9tp/cluster-candidate-5-0-npd8xs should provision Unexpected error: <fmt.wrapError \| 0xc00104eb20>: failed to create HCP cluster cluster-candidate-5-0-npd8xs: failed waiting for cluster="cluster-candidate-5-0-npd8xs" in resourcegroup="rg-candidate-5-0-krw7bcv4h9tp" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/677c6d18-0a88-45ea-8eef-dcf3bcaf9526 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/677c6d18-0a88-45ea-8eef-dcf3bcaf9526", "name": "677c6d18-0a88-45ea-8eef-dcf3bcaf9526", "status": "Failed", "startTime": "2026-09-22T13:32:01.537906113Z", "endTime": "2026-09-22T14:01:04.991194141Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc0005747c0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-f5kx8sntpz47" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bfe16b2b-1ba0-4224-a47a-10f0686acafc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bfe16b2b-1ba0-4224-a47a-10f0686acafc", "name": "bfe16b2b-1ba0-4224-a47a-10f0686acafc", "status": "Failed", "startTime": "2026-09-22T13:33:57.252045262Z", "endTime": "2026-09-22T14:03:04.849972903Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc0005747c0>: failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-f5kx8sntpz47" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bfe16b2b-1ba0-4224-a47a-10f0686acafc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bfe16b2b-1ba0-4224-a47a-10f0686acafc", "name": "bfe16b2b-1ba0-4224-a47a-10f0686acafc", "status": "Failed", "startTime": "2026-09-22T13:33:57.252045262Z", "endTime": "2026-09-22T14:03:04.849972903Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated; provider Microsoft.RedHatOpenShift 12:46:33 UTC · failed at e2e · test SRE should be able to log into a cluster via a breakglass session · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000823be0>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-7p9grtkmqq95" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1340bd4e-1432-48c6-b9c6-1e08cde4b773 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1340bd4e-1432-48c6-b9c6-1e08cde4b773", "name": "1340bd4e-1432-48c6-b9c6-1e08cde4b773", "status": "Failed", "startTime": "2026-09-22T13:32:04.276124571Z", "endTime": "2026-09-22T14:01:04.883826856Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:92]: failed to create HCP cluster "sre-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000823be0>: failed to create HCP cluster sre-hcp-cluster: failed waiting for cluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-7p9grtkmqq95" to finish creating: GET https://rp.j7129984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1340bd4e-1432-48c6-b9c6-1e08cde4b773 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/1340bd4e-1432-48c6-b9c6-1e08cde4b773", "name": "1340bd4e-1432-48c6-b9c6-1e08cde4b773", "status": "Failed", "startTime": "2026-09-22T13:32:04.276124571Z", "endTime": "2026-09-22T14:01:04.883826856Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [hypershiftHostedCluster] ReadDesire has no kube content; [servingCABundle] ServingCABundle not yet populated" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [svc] userJourneyClusterProvisionErrors1h5m fired 12:46:33 UTC · failed at alert · test [aro-hcp-observability] [svc] alert userJourneyClusterProvisionErrors1h5m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T14:07:46Z Severity: Sev3 Labels: alertname="userJourneyClusterProvisionErrors1h5m", cluster="ci01-j7129984-svc", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7129984/providers/microsoft.monitor/accounts/services-j7129984", region="westus3", severity="3", short_window="5m", slo="cluster-provision-errors" Description: More than 72% of completed cluster create (install) operations on ci01-j7129984-svc failed over the last hour with at least 3 failures, a 14.4x burn of the 95% SLO budget. alert [svc] userJourneyClusterProvisionErrors6h30m fired 12:46:33 UTC · failed at alert · test [aro-hcp-observability] [svc] alert userJourneyClusterProvisionErrors6h30m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T14:32:47Z Severity: Sev3 Labels: alertname="userJourneyClusterProvisionErrors6h30m", cluster="ci01-j7129984-svc", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7129984/providers/microsoft.monitor/accounts/services-j7129984", region="westus3", severity="3", short_window="30m", slo="cluster-provision-errors" Description: More than 30% of completed cluster create (install) operations on ci01-j7129984-svc failed over the last 6 hours (at least 5 completions), a 6x burn of the 95% SLO budget.
12:15:42	2h 13m 22s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head c530dc6	westus3	alert, e2e	failed	#6364 (open)	4	e2e (2) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift 12:15:42 UTC · failed at e2e · test Customer should be able to retrieve cluster and nodepool status active versions · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:88]: failed to create HCP cluster "v20261001" via v20261001preview Unexpected error: <fmt.wrapError \| 0xc000fb01e0>: failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-22ntvrtddssz" to finish creating: GET https://rp.j8940032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/242f24a5-8f64-4654-9b39-e30267e94ed1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/242f24a5-8f64-4654-9b39-e30267e94ed1", "name": "242f24a5-8f64-4654-9b39-e30267e94ed1", "status": "Failed", "startTime": "2026-09-22T13:01:52.830008497Z", "endTime": "2026-09-22T13:30:57.820386496Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 21m40s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:88]: failed to create HCP cluster "v20261001" via v20261001preview Unexpected error: <fmt.wrapError \| 0xc000fb01e0>: failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-22ntvrtddssz" to finish creating: GET https://rp.j8940032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/242f24a5-8f64-4654-9b39-e30267e94ed1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/242f24a5-8f64-4654-9b39-e30267e94ed1", "name": "242f24a5-8f64-4654-9b39-e30267e94ed1", "status": "Failed", "startTime": "2026-09-22T13:01:52.830008497Z", "endTime": "2026-09-22T13:30:57.820386496Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 21m40s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 12:15:42 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error: <errors.joinError \| 0xc001b1a0a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-wwxhmhlsh7vt" to finish deleting: GET https://rp.j8940032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bacc4e5a-7f20-4127-8aed-f8e207759e95 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bacc4e5a-7f20-4127-8aed-f8e207759e95", "name": "bacc4e5a-7f20-4127-8aed-f8e207759e95", "status": "Failed", "startTime": "2026-09-22T13:46:58.861939308Z", "endTime": "2026-09-22T14:11:01.43159062Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t02sdao0ej27cdpe8l00d8ep8hru718 still exists (deletion dispatched at 2026-09-22T13:47:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error: <errors.joinError \| 0xc001b1a0a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-wwxhmhlsh7vt" to finish deleting: GET https://rp.j8940032.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bacc4e5a-7f20-4127-8aed-f8e207759e95 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bacc4e5a-7f20-4127-8aed-f8e207759e95", "name": "bacc4e5a-7f20-4127-8aed-f8e207759e95", "status": "Failed", "startTime": "2026-09-22T13:46:58.861939308Z", "endTime": "2026-09-22T14:11:01.43159062Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t02sdao0ej27cdpe8l00d8ep8hru718 still exists (deletion dispatched at 2026-09-22T13:47:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [hcp] KubePodNotReady fired 12:15:42 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests Show raw failure alert fired 4 time(s) (1 unknown, 3 known) Firing 1: State: Resolved Started: 2026-09-22T13:15:15Z Ended: 2026-09-22T13:39:12Z Severity: Sev3 Labels: alertname="KubePodNotReady", cluster="ci01-j8940032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t02sh3cjhqhk5sqr2356elqhegnu68u-v20261001", pod="router-6c76dd549c-ds2wz", region="westus3", severity="warning" Description: Pod ocm-arohcpci01-2t02sh3cjhqhk5sqr2356elqhegnu68u-v20261001/router-6c76dd549c-ds2wz has been in a non-ready state for longer than 5 minutes. alert [svc] FrontendPathLatency fired 12:15:42 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T13:50:49Z Ended: 2026-09-22T14:01:52Z Severity: Sev3 Labels: alertname="FrontendPathLatency", cluster="ci01-j8940032-svc", component="frontend", method="delete", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: More than 5% of frontend requests for delete /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j8940032-svc).
12:15:38	1h 59m 59s	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 618b096 · merge c83bdab	canadacentral	n/a	passed	#7009 (merged)	0	n/a
12:00:30	2h 13m 13s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 95f6c7a · merge 95f6c7a	westus3	alert, e2e	failed	#6948 (merged)	3	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 12:00:30 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000c88840>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-784f6vzvhd54" to finish deleting: GET https://rp.j1951488.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/22b142d3-4083-40a9-b2f7-0c8d747fb92b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/22b142d3-4083-40a9-b2f7-0c8d747fb92b", "name": "22b142d3-4083-40a9-b2f7-0c8d747fb92b", "status": "Failed", "startTime": "2026-09-22T13:22:36.19366526Z", "endTime": "2026-09-22T13:46:43.924656305Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t02o05k4mjacdppm5v91uu7di2s60fh still exists (deletion dispatched at 2026-09-22T13:22:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000c88840>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-784f6vzvhd54" to finish deleting: GET https://rp.j1951488.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/22b142d3-4083-40a9-b2f7-0c8d747fb92b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/22b142d3-4083-40a9-b2f7-0c8d747fb92b", "name": "22b142d3-4083-40a9-b2f7-0c8d747fb92b", "status": "Failed", "startTime": "2026-09-22T13:22:36.19366526Z", "endTime": "2026-09-22T13:46:43.924656305Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t02o05k4mjacdppm5v91uu7di2s60fh still exists (deletion dispatched at 2026-09-22T13:22:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [svc] FrontendPathLatency fired 12:00:30 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T13:27:35Z Ended: 2026-09-22T13:40:37Z Severity: Sev3 Labels: alertname="FrontendPathLatency", cluster="ci01-j1951488-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning" Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j1951488-svc). alert [svc] ServiceMemoryDrift fired 12:00:30 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T13:57:41Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j1951488-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-hzln7", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-hzln7 (namespace arobit) on cluster ci01-j1951488-svc has averaged 126.1% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
12:00:29	2h 20m 42s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 170f768	centralus	alert, e2e	failed	#6128 (open)	4	e2e (2) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 12:00:29 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000bc0210>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-lpmmxlnr5f8b" to finish deleting: GET https://rp.j7674624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4d28fedd-a09c-4916-9a5b-65e9ecdf6442 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4d28fedd-a09c-4916-9a5b-65e9ecdf6442", "name": "4d28fedd-a09c-4916-9a5b-65e9ecdf6442", "status": "Failed", "startTime": "2026-09-22T13:39:34.975442985Z", "endTime": "2026-09-22T14:03:35.724040601Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t02n2vj0cpu1967qjmj20dbkb8pajv6 still exists (deletion dispatched at 2026-09-22T13:39:35Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000bc0210>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-lpmmxlnr5f8b" to finish deleting: GET https://rp.j7674624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4d28fedd-a09c-4916-9a5b-65e9ecdf6442 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/4d28fedd-a09c-4916-9a5b-65e9ecdf6442", "name": "4d28fedd-a09c-4916-9a5b-65e9ecdf6442", "status": "Failed", "startTime": "2026-09-22T13:39:34.975442985Z", "endTime": "2026-09-22T14:03:35.724040601Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t02n2vj0cpu1967qjmj20dbkb8pajv6 still exists (deletion dispatched at 2026-09-22T13:39:35Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 12:00:29 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a86060>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-21-dqw4tg" in resourcegroup="rg-cp-ystream-upgrade-4-21-rpz8pwqkf6x2" to finish deleting: GET https://rp.j7674624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/59fe04db-687e-4883-94b3-d71e6b0e6949 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/59fe04db-687e-4883-94b3-d71e6b0e6949", "name": "59fe04db-687e-4883-94b3-d71e6b0e6949", "status": "Failed", "startTime": "2026-09-22T13:23:25.69462366Z", "endTime": "2026-09-22T13:47:29.022299266Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t02qcvn03mmp1b6p4cv8m8otqvf9i4c still exists (deletion dispatched at 2026-09-22T13:23:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a86060>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-21-dqw4tg" in resourcegroup="rg-cp-ystream-upgrade-4-21-rpz8pwqkf6x2" to finish deleting: GET https://rp.j7674624.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/59fe04db-687e-4883-94b3-d71e6b0e6949 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/59fe04db-687e-4883-94b3-d71e6b0e6949", "name": "59fe04db-687e-4883-94b3-d71e6b0e6949", "status": "Failed", "startTime": "2026-09-22T13:23:25.69462366Z", "endTime": "2026-09-22T13:47:29.022299266Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t02qcvn03mmp1b6p4cv8m8otqvf9i4c still exists (deletion dispatched at 2026-09-22T13:23:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [svc] FrontendPathMedianLatency fired 12:00:29 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T13:20:51Z Ended: 2026-09-22T13:30:54Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j7674624-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j7674624-svc). alert [svc] ServiceMemoryDrift fired 12:00:29 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T13:59:56Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j7674624-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-8gnzs", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-8gnzs (namespace arobit) on cluster ci01-j7674624-svc has averaged 122.9% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
12:00:21	2h 14m 27s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 54049e5 · merge 54049e5	centralus	e2e	failed	#6995 (merged)	1	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 12:00:21 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00186a078>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-d6952r" in resourcegroup="rg-cp-ystream-upgrade-4-22-2drztfzcz8ln" to finish deleting: GET https://rp.j5061120.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2b6d8126-6b4c-4de8-86ce-6d951164492f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2b6d8126-6b4c-4de8-86ce-6d951164492f", "name": "2b6d8126-6b4c-4de8-86ce-6d951164492f", "status": "Failed", "startTime": "2026-09-22T13:23:15.903664501Z", "endTime": "2026-09-22T13:47:15.971767798Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t02nnilq7ncv48hc3nk14tuqve9oq7s still exists (deletion dispatched at 2026-09-22T13:23:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00186a078>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-d6952r" in resourcegroup="rg-cp-ystream-upgrade-4-22-2drztfzcz8ln" to finish deleting: GET https://rp.j5061120.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2b6d8126-6b4c-4de8-86ce-6d951164492f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2b6d8126-6b4c-4de8-86ce-6d951164492f", "name": "2b6d8126-6b4c-4de8-86ce-6d951164492f", "status": "Failed", "startTime": "2026-09-22T13:23:15.903664501Z", "endTime": "2026-09-22T13:47:15.971767798Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t02nnilq7ncv48hc3nk14tuqve9oq7s still exists (deletion dispatched at 2026-09-22T13:23:16Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
12:00:18	2h 5m 42s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head a65a65d	centralus	alert, e2e	failed	#6756 (open)	3	e2e (2) ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 12:00:18 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster Unexpected error: <fmt.wrapError \| 0xc000b45e20>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nvlgx5jfn6c7" to finish creating: GET https://rp.j2105984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/59212393-2ed7-490f-a765-56b81be64c8e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/59212393-2ed7-490f-a765-56b81be64c8e", "name": "59212393-2ed7-490f-a765-56b81be64c8e", "status": "Failed", "startTime": "2026-09-22T12:57:41.103155101Z", "endTime": "2026-09-22T13:16:45.486929869Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster Unexpected error: <fmt.wrapError \| 0xc000b45e20>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nvlgx5jfn6c7" to finish creating: GET https://rp.j2105984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/59212393-2ed7-490f-a765-56b81be64c8e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/59212393-2ed7-490f-a765-56b81be64c8e", "name": "59212393-2ed7-490f-a765-56b81be64c8e", "status": "Failed", "startTime": "2026-09-22T12:57:41.103155101Z", "endTime": "2026-09-22T13:16:45.486929869Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 12:00:18 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000a8a4e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-4nfw84mzx8tm" to finish deleting: GET https://rp.j2105984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b61d39c7-956d-42aa-8f64-5073f11a84af -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b61d39c7-956d-42aa-8f64-5073f11a84af", "name": "b61d39c7-956d-42aa-8f64-5073f11a84af", "status": "Failed", "startTime": "2026-09-22T13:22:25.901102704Z", "endTime": "2026-09-22T13:46:32.743102747Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t02mfa1v5gfqso6k7vcuahuapgr4h0p still exists (deletion dispatched at 2026-09-22T13:22:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000a8a4e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-4nfw84mzx8tm" to finish deleting: GET https://rp.j2105984.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b61d39c7-956d-42aa-8f64-5073f11a84af -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b61d39c7-956d-42aa-8f64-5073f11a84af", "name": "b61d39c7-956d-42aa-8f64-5073f11a84af", "status": "Failed", "startTime": "2026-09-22T13:22:25.901102704Z", "endTime": "2026-09-22T13:46:32.743102747Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t02mfa1v5gfqso6k7vcuahuapgr4h0p still exists (deletion dispatched at 2026-09-22T13:22:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] FrontendPathMedianLatency fired 12:00:18 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T12:56:44Z Ended: 2026-09-22T13:29:42Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j2105984-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j2105984-svc).
11:57:46	1h 55m 53s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 21a1472	canadacentral	e2e	failed	#6993 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 11:57:46 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np Unexpected error: <fmt.wrapError \| 0xc000e9f240>: failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-gmkmxkr8rtmb" to finish creating: GET https://rp.j8670976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/24481b8a-04e4-437a-8420-e00fa62a404d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/24481b8a-04e4-437a-8420-e00fa62a404d", "name": "24481b8a-04e4-437a-8420-e00fa62a404d", "status": "Failed", "startTime": "2026-09-22T13:09:58.667800577Z", "endTime": "2026-09-22T13:29:05.886575623Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np Unexpected error: <fmt.wrapError \| 0xc000e9f240>: failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-gmkmxkr8rtmb" to finish creating: GET https://rp.j8670976.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/24481b8a-04e4-437a-8420-e00fa62a404d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/24481b8a-04e4-437a-8420-e00fa62a404d", "name": "24481b8a-04e4-437a-8420-e00fa62a404d", "status": "Failed", "startTime": "2026-09-22T13:09:58.667800577Z", "endTime": "2026-09-22T13:29:05.886575623Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred
11:56:22	55m 14s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 17723dd	centralus	provision	failed	⚠#6590 (open)	1	provision (1) ERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardDdsv6Family for region centralus. If you want to increase the quota, please follow this instruction: <url> Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.; provider Microsoft.ContainerService 11:56:22 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2) · suite templatize-pipeline Show raw failure time=2026-09-22T12:18:17.689Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n Kind: Shell\n Command: ./aks-cluster-create\n" time=2026-09-22T12:45:29.994Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-22T12:18:17.695906444Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-22T12:18:20.00208003Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-22T12:18:20.14445826Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-22T12:45:29.992055155Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"CF0FD6F1065F90B5\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j2830976-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T12:43:18.8605679Z\",\"duration\":\"PT8.5695733S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 44 for family StandardDdsv6Family for region centralus. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T12:45:29.992133756Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"DD6EAF8731EA851B\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j2830976-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T12:43:18.8605679Z\",\"duration\":\"PT1M23.5357859S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T12:45:29.992147237Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"9E2ABCCAB49D4C81\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j2830976-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T12:43:18.8605679Z\",\"duration\":\"PT1M23.3548981S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T12:45:29.992157287Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"42D7531EF6508071\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j2830976-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T12:43:18.8605679Z\",\"duration\":\"PT1M53.0336442S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T12:45:29.992166697Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"1B516DD7916AE405\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j2830976-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T12:18:26.1875166Z\",\"duration\":\"PT26M47.0458862S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T12:45:29.992178797Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"2569996FCD48888B\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2\",\"resourceName\":\"ci01-j2830976-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T12:18:26.1875166Z\",\"duration\":\"PT24M52.6018784S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T12:45:29.992201978Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2830976-mgmt-2\",\"clusterName\":\"ci01-j2830976-mgmt-2\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584115273851533055\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"status\\\": \\\"Failed\\\",\\n \\\"error\\\": {\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n \\\"details\\\": [\\n {\\n \\\"code\\\": \\\"BadRequest\\\",\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2830976-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2830976-mgmt-2/agentPools/userswft3\\\",\\n \\\"message\\\": \\\"{\\\\r\\\\n \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n \\\\\\\"details\\\\\\\": null,\\\\r\\\\n \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 44 for family StandardDdsv6Family for region centralus. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n }\\n ]\\n }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"27m12.296304024s\"}\n exit status 1"
11:29:20	2h 18m 59s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 10d1e6e · merge 10d1e6e	westus3	alert, e2e	failed	#7070 (merged)	2	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 11:29:20 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00104c078>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-hj7wxhq2hq8n" to finish deleting: GET https://rp.j5129728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d29e61b2-26f8-48f8-8b4e-5d1a334405b3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d29e61b2-26f8-48f8-8b4e-5d1a334405b3", "name": "d29e61b2-26f8-48f8-8b4e-5d1a334405b3", "status": "Failed", "startTime": "2026-09-22T13:01:37.902956089Z", "endTime": "2026-09-22T13:25:46.268749052Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t02717lssolbvgogntqqtclnqlr0fpl still exists (deletion dispatched at 2026-09-22T13:01:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00104c078>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-hj7wxhq2hq8n" to finish deleting: GET https://rp.j5129728.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d29e61b2-26f8-48f8-8b4e-5d1a334405b3 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/d29e61b2-26f8-48f8-8b4e-5d1a334405b3", "name": "d29e61b2-26f8-48f8-8b4e-5d1a334405b3", "status": "Failed", "startTime": "2026-09-22T13:01:37.902956089Z", "endTime": "2026-09-22T13:25:46.268749052Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t02717lssolbvgogntqqtclnqlr0fpl still exists (deletion dispatched at 2026-09-22T13:01:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] ServiceMemoryDrift fired 11:29:20 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T13:26:35Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j5129728-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-7wvxd", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-7wvxd (namespace arobit) on cluster ci01-j5129728-svc has averaged 126.1% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
11:23:29	2h 13m 54s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head d433c21 · merge d433c21	centralus	alert, e2e	failed	#6752 (merged)	3	e2e (2) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 11:23:29 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00190c378>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-sfzxvlgpv6ph" to finish deleting: GET https://rp.j8737152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d32a2314-3299-439e-8c20-7c7037b11098 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d32a2314-3299-439e-8c20-7c7037b11098", "name": "d32a2314-3299-439e-8c20-7c7037b11098", "status": "Failed", "startTime": "2026-09-22T12:34:28.832612498Z", "endTime": "2026-09-22T12:58:35.458664282Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t022ukf70n48f1pegs95hv1qcaeki9p still exists (deletion dispatched at 2026-09-22T12:34:36Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00190c378>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-sfzxvlgpv6ph" to finish deleting: GET https://rp.j8737152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d32a2314-3299-439e-8c20-7c7037b11098 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/d32a2314-3299-439e-8c20-7c7037b11098", "name": "d32a2314-3299-439e-8c20-7c7037b11098", "status": "Failed", "startTime": "2026-09-22T12:34:28.832612498Z", "endTime": "2026-09-22T12:58:35.458664282Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t022ukf70n48f1pegs95hv1qcaeki9p still exists (deletion dispatched at 2026-09-22T12:34:36Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 11:23:29 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000012108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-65gqgkqpjknr" to finish deleting: GET https://rp.j8737152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b89e8d10-3c6f-47c2-b094-8d090e0e8717 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b89e8d10-3c6f-47c2-b094-8d090e0e8717", "name": "b89e8d10-3c6f-47c2-b094-8d090e0e8717", "status": "Failed", "startTime": "2026-09-22T12:54:20.279157709Z", "endTime": "2026-09-22T13:18:21.263130823Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t022sgna103g4j62q4jks87krqaalg9 still exists (deletion dispatched at 2026-09-22T12:54:21Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000012108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-65gqgkqpjknr" to finish deleting: GET https://rp.j8737152.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b89e8d10-3c6f-47c2-b094-8d090e0e8717 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b89e8d10-3c6f-47c2-b094-8d090e0e8717", "name": "b89e8d10-3c6f-47c2-b094-8d090e0e8717", "status": "Failed", "startTime": "2026-09-22T12:54:20.279157709Z", "endTime": "2026-09-22T13:18:21.263130823Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t022sgna103g4j62q4jks87krqaalg9 still exists (deletion dispatched at 2026-09-22T12:54:21Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] ServiceMemoryDrift fired 11:23:29 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T13:14:30Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j8737152-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-7549w", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-7549w (namespace arobit) on cluster ci01-j8737152-svc has averaged 126% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
11:04:56	2h 4m 56s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 989af9a	canadacentral	e2e	failed	#6092 (open)	2	e2e (2) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 11:04:56 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0012202d0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-nj95pp" in resourcegroup="rg-zstream-upgrade-4-20-zxqfccsdzw5x" to finish deleting: GET https://rp.j7511040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ffa6556f-e169-447b-ba03-2ec23b8ebfcf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ffa6556f-e169-447b-ba03-2ec23b8ebfcf", "name": "ffa6556f-e169-447b-ba03-2ec23b8ebfcf", "status": "Failed", "startTime": "2026-09-22T12:26:07.876024314Z", "endTime": "2026-09-22T12:50:12.128002699Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t01u4bmgb6jfsk48mi3c59g61dc3ngm still exists (deletion dispatched at 2026-09-22T12:26:09Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0012202d0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-nj95pp" in resourcegroup="rg-zstream-upgrade-4-20-zxqfccsdzw5x" to finish deleting: GET https://rp.j7511040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ffa6556f-e169-447b-ba03-2ec23b8ebfcf -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/ffa6556f-e169-447b-ba03-2ec23b8ebfcf", "name": "ffa6556f-e169-447b-ba03-2ec23b8ebfcf", "status": "Failed", "startTime": "2026-09-22T12:26:07.876024314Z", "endTime": "2026-09-22T12:50:12.128002699Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t01u4bmgb6jfsk48mi3c59g61dc3ngm still exists (deletion dispatched at 2026-09-22T12:26:09Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: ResourceNotFound; detail message The resource 'locations/hcpOperationStatuses/<uuid>' under resource group '' was not found.; provider Microsoft.RedHatOpenShift 11:04:56 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:329]: failed to get admin REST config for cluster np-noedge-7z925l Unexpected error: <fmt.wrapError \| 0xc00077d780>: failed waiting for hcpCluster="np-noedge-7z925l" in resourcegroup="rg-np-noedge-7z925l-xwhx4zwbzvsx" to finish getting creds: GET https://rp.j7511040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ac55f4d-25b7-4531-81f3-de46161013d9 -------------------------------------------------------------------------------- RESPONSE 404: 404 Not Found ERROR CODE: ResourceNotFound -------------------------------------------------------------------------------- { "error": { "code": "ResourceNotFound", "message": "The resource 'locations/hcpOperationStatuses/8ac55f4d-25b7-4531-81f3-de46161013d9' under resource group '' was not found.", "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ac55f4d-25b7-4531-81f3-de46161013d9" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:329]: failed to get admin REST config for cluster np-noedge-7z925l Unexpected error: <fmt.wrapError \| 0xc00077d780>: failed waiting for hcpCluster="np-noedge-7z925l" in resourcegroup="rg-np-noedge-7z925l-xwhx4zwbzvsx" to finish getting creds: GET https://rp.j7511040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ac55f4d-25b7-4531-81f3-de46161013d9 -------------------------------------------------------------------------------- RESPONSE 404: 404 Not Found ERROR CODE: ResourceNotFound -------------------------------------------------------------------------------- { "error": { "code": "ResourceNotFound", "message": "The resource 'locations/hcpOperationStatuses/8ac55f4d-25b7-4531-81f3-de46161013d9' under resource group '' was not found.", "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ac55f4d-25b7-4531-81f3-de46161013d9" } } -------------------------------------------------------------------------------- ... occurred
11:02:52	2h 19m 46s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 87e40a0	centralus	e2e	failed	#6922 (open)	4	e2e (4) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 11:02:52 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00129c048>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-rwcrtszf7jfv" to finish deleting: GET https://rp.j4782848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6c98f5ee-3d3e-4d06-adb1-93b02a59b699 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6c98f5ee-3d3e-4d06-adb1-93b02a59b699", "name": "6c98f5ee-3d3e-4d06-adb1-93b02a59b699", "status": "Failed", "startTime": "2026-09-22T12:40:11.897115045Z", "endTime": "2026-09-22T13:04:14.128066943Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t01t2m7o942c4sn0eema6t828rb1f1u still exists (deletion dispatched at 2026-09-22T12:40:14Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00129c048>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-rwcrtszf7jfv" to finish deleting: GET https://rp.j4782848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6c98f5ee-3d3e-4d06-adb1-93b02a59b699 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6c98f5ee-3d3e-4d06-adb1-93b02a59b699", "name": "6c98f5ee-3d3e-4d06-adb1-93b02a59b699", "status": "Failed", "startTime": "2026-09-22T12:40:11.897115045Z", "endTime": "2026-09-22T13:04:14.128066943Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t01t2m7o942c4sn0eema6t828rb1f1u still exists (deletion dispatched at 2026-09-22T12:40:14Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 11:02:52 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000d48120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-s5gwd7" in resourcegroup="rg-cp-ystream-upgrade-4-22-4crbvttx4ddn" to finish deleting: GET https://rp.j4782848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/efee312f-62e1-4b0e-aca7-6c7e60974ce7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/efee312f-62e1-4b0e-aca7-6c7e60974ce7", "name": "efee312f-62e1-4b0e-aca7-6c7e60974ce7", "status": "Failed", "startTime": "2026-09-22T12:24:03.210379981Z", "endTime": "2026-09-22T12:48:03.095196437Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t01t0n3nhj5mkrp4pcjj6062lg3lv8o still exists (deletion dispatched at 2026-09-22T12:24:04Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000d48120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-s5gwd7" in resourcegroup="rg-cp-ystream-upgrade-4-22-4crbvttx4ddn" to finish deleting: GET https://rp.j4782848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/efee312f-62e1-4b0e-aca7-6c7e60974ce7 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/efee312f-62e1-4b0e-aca7-6c7e60974ce7", "name": "efee312f-62e1-4b0e-aca7-6c7e60974ce7", "status": "Failed", "startTime": "2026-09-22T12:24:03.210379981Z", "endTime": "2026-09-22T12:48:03.095196437Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t01t0n3nhj5mkrp4pcjj6062lg3lv8o still exists (deletion dispatched at 2026-09-22T12:24:04Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 11:02:52 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001b50378>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-87wgx5" in resourcegroup="rg-np-version-upgrade-87wgx5-pcmbclp6gkrs" to finish deleting: GET https://rp.j4782848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2d459377-0fed-4f30-8665-439104478418 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2d459377-0fed-4f30-8665-439104478418", "name": "2d459377-0fed-4f30-8665-439104478418", "status": "Failed", "startTime": "2026-09-22T12:33:05.196504994Z", "endTime": "2026-09-22T12:57:14.101136035Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t01t0nogib9ejpsse48iaemplhqusib still exists (deletion dispatched at 2026-09-22T12:33:06Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001b50378>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-87wgx5" in resourcegroup="rg-np-version-upgrade-87wgx5-pcmbclp6gkrs" to finish deleting: GET https://rp.j4782848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2d459377-0fed-4f30-8665-439104478418 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/2d459377-0fed-4f30-8665-439104478418", "name": "2d459377-0fed-4f30-8665-439104478418", "status": "Failed", "startTime": "2026-09-22T12:33:05.196504994Z", "endTime": "2026-09-22T12:57:14.101136035Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t01t0nogib9ejpsse48iaemplhqusib still exists (deletion dispatched at 2026-09-22T12:33:06Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 11:02:52 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001990210>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-bvfmtf6bf2br" to finish deleting: GET https://rp.j4782848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6d2c27ff-6119-47a9-b767-c54c12209ed2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6d2c27ff-6119-47a9-b767-c54c12209ed2", "name": "6d2c27ff-6119-47a9-b767-c54c12209ed2", "status": "Failed", "startTime": "2026-09-22T12:20:47.737752758Z", "endTime": "2026-09-22T12:44:53.098039041Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t01sq695pk0nqm2fvfsrqu4l67jje6m still exists (deletion dispatched at 2026-09-22T12:20:49Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001990210>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-bvfmtf6bf2br" to finish deleting: GET https://rp.j4782848.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6d2c27ff-6119-47a9-b767-c54c12209ed2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6d2c27ff-6119-47a9-b767-c54c12209ed2", "name": "6d2c27ff-6119-47a9-b767-c54c12209ed2", "status": "Failed", "startTime": "2026-09-22T12:20:47.737752758Z", "endTime": "2026-09-22T12:44:53.098039041Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t01sq695pk0nqm2fvfsrqu4l67jje6m still exists (deletion dispatched at 2026-09-22T12:20:49Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
10:58:09	1h 0m 55s	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 618b096 · merge c83bdab	canadacentral	provision	failed	⚠#7009 (merged)	1	provision (1) error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j8724736-mgmt-2","clusterName":"ci01-j872473 10:58:09 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2) · suite templatize-pipeline Show raw failure time=2026-09-22T11:23:53.670Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n Kind: Shell\n Command: ./aks-cluster-create\n" time=2026-09-22T11:53:53.299Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-22T11:23:53.695872086Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8724736-mgmt-2\",\"clusterName\":\"ci01-j8724736-mgmt-2\",\"region\":\"canadacentral\"}\n{\"time\":\"2026-09-22T11:23:55.867241647Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8724736-mgmt-2\",\"clusterName\":\"ci01-j8724736-mgmt-2\",\"region\":\"canadacentral\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-22T11:23:55.995607642Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j8724736-mgmt-2\",\"clusterName\":\"ci01-j8724736-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"
10:11:43	2h 9m 49s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 8ae5d54 · merge 8ae5d54	centralus	e2e	failed	#7067 (merged)	1	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 10:11:43 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00106c0a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-8jmmzcb9t2nx" to finish deleting: GET https://rp.j2185856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f4c77cde-689d-4991-88ee-14b0517d1495 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f4c77cde-689d-4991-88ee-14b0517d1495", "name": "f4c77cde-689d-4991-88ee-14b0517d1495", "status": "Failed", "startTime": "2026-09-22T11:38:59.29364075Z", "endTime": "2026-09-22T12:03:02.197932791Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t010oqmj8rpakhnf6q2d5om47fbernn still exists (deletion dispatched at 2026-09-22T11:39:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00106c0a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-8jmmzcb9t2nx" to finish deleting: GET https://rp.j2185856.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f4c77cde-689d-4991-88ee-14b0517d1495 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f4c77cde-689d-4991-88ee-14b0517d1495", "name": "f4c77cde-689d-4991-88ee-14b0517d1495", "status": "Failed", "startTime": "2026-09-22T11:38:59.29364075Z", "endTime": "2026-09-22T12:03:02.197932791Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t010oqmj8rpakhnf6q2d5om47fbernn still exists (deletion dispatched at 2026-09-22T11:39:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
10:04:32	44m 48s	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 618b096 · merge c83bdab	canadacentral	provision	failed	★#7009 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: <url> Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.; provider Microsoft.ContainerService 10:04:32 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2) · suite templatize-pipeline Show raw failure time=2026-09-22T10:35:24.276Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n Kind: Shell\n Command: ./aks-cluster-create\n" time=2026-09-22T10:42:31.669Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-22T10:35:24.282791483Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\"}\n{\"time\":\"2026-09-22T10:35:26.284841392Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-22T10:35:26.448708771Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-22T10:42:31.666852933Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"48630ADF8315F9F4\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j2444800-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T10:40:38.6323694Z\",\"duration\":\"PT1M40.6270261S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T10:42:31.666918654Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"50E367DC062B740B\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j2444800-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T10:40:38.6323694Z\",\"duration\":\"PT1M39.3479577S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T10:42:31.666940194Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"96EF32884A17F314\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j2444800-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T10:40:38.6323694Z\",\"duration\":\"PT4.1058994S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T10:42:31.666978395Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"1E29754134C2A673\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j2444800-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T10:35:30.2867656Z\",\"duration\":\"PT5M11.8733849S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T10:42:31.667019736Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"DD00B42016EFC1CA\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j2444800-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T10:40:38.6323694Z\",\"duration\":\"PT3.4027868S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T10:42:31.667063427Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"3962D5B17452A0C8\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2\",\"resourceName\":\"ci01-j2444800-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T10:35:30.2867656Z\",\"duration\":\"PT5M8.2796787S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T10:42:31.667099327Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j2444800-mgmt-2\",\"clusterName\":\"ci01-j2444800-mgmt-2\",\"region\":\"canadacentral\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584115335588388698\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"status\\\": \\\"Failed\\\",\\n \\\"error\\\": {\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n \\\"details\\\": [\\n {\\n \\\"code\\\": \\\"BadRequest\\\",\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2/agentPools/userswft2\\\",\\n \\\"message\\\": \\\"{\\\\r\\\\n \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n \\\\\\\"details\\\\\\\": null,\\\\r\\\\n \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n },\\n {\\n \\\"code\\\": \\\"BadRequest\\\",\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2/agentPools/userswft1\\\",\\n \\\"message\\\": \\\"{\\\\r\\\\n \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n \\\\\\\"details\\\\\\\": null,\\\\r\\\\n \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n },\\n {\\n \\\"code\\\": \\\"BadRequest\\\",\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j2444800-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j2444800-mgmt-2/agentPools/userswft3\\\",\\n \\\"message\\\": \\\"{\\\\r\\\\n \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n \\\\\\\"details\\\\\\\": null,\\\\r\\\\n \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 0 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n }\\n ]\\n }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m7.384317605s\"}\n exit status 1"
10:01:02	2h 27m 17s	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head eca3647 · merge 23179a5	westus3	alert, e2e	failed	★#7066 (merged)	6	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 10:01:02 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001130b58>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-c9mwnv9mdv9l" to finish deleting: GET https://rp.j2193024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2b236a4e-5c28-455a-a3b8-408d1c8a5832 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2b236a4e-5c28-455a-a3b8-408d1c8a5832", "name": "2b236a4e-5c28-455a-a3b8-408d1c8a5832", "status": "Failed", "startTime": "2026-09-22T11:43:46.417280075Z", "endTime": "2026-09-22T12:07:51.315583112Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t012ocf0lcq23fakt2qrpvp9htm8g26 still exists (deletion dispatched at 2026-09-22T11:43:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001130b58>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-c9mwnv9mdv9l" to finish deleting: GET https://rp.j2193024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2b236a4e-5c28-455a-a3b8-408d1c8a5832 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2b236a4e-5c28-455a-a3b8-408d1c8a5832", "name": "2b236a4e-5c28-455a-a3b8-408d1c8a5832", "status": "Failed", "startTime": "2026-09-22T11:43:46.417280075Z", "endTime": "2026-09-22T12:07:51.315583112Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t012ocf0lcq23fakt2qrpvp9htm8g26 still exists (deletion dispatched at 2026-09-22T11:43:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (5) alert [svc] FrontendPathMedianLatency fired 10:01:02 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T11:05:13Z Ended: 2026-09-22T11:38:14Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j2193024-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j2193024-svc). alert [svc] ServiceMemoryDrift fired 10:01:02 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T12:10:47Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j2193024-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-nffxd", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-nffxd (namespace arobit) on cluster ci01-j2193024-svc has averaged 126.2% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml. alert [svc] userJourneyFrontendLatency6h30m fired 10:01:02 UTC · failed at alert · test [aro-hcp-observability] [svc] alert userJourneyFrontendLatency6h30m does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T11:40:32Z Severity: Sev3 Labels: alertname="userJourneyFrontendLatency6h30m", cluster="ci01-j2193024-svc", component="slo", long_window="6h", region="westus3", severity="3", short_window="30m", slo="frontend-latency" Description: More than 6% of Frontend requests on cluster ci01-j2193024-svc exceeded 1s on both the 6h and 30m windows (6x burn of the 1% latency budget). alert [svc] BackendControllerQueueDepthHigh fired 10:01:02 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-09-22T11:14:13Z Ended: 2026-09-22T11:26:09Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2193024-svc", component="backend", name="clusterresources", region="westus3", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. Firing 2: State: Resolved Started: 2026-09-22T11:24:10Z Ended: 2026-09-22T11:34:10Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2193024-svc", component="backend", name="csstatedump", region="westus3", severity="warning" Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed. alert [svc] FrontendPathLatency fired 10:01:02 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T11:41:13Z Ended: 2026-09-22T11:50:11Z Severity: Sev3 Labels: alertname="FrontendPathLatency", cluster="ci01-j2193024-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters", severity="warning" Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters exceeded 1 second over the past 30 minutes (cluster ci01-j2193024-svc).
10:01:00	2h 25m 12s	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 6c5cfd0 · merge a0bf660	centralus	e2e	failed	#6503 (merged)	1	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 10:01:00 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001234210>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-7lcrh4lksx7n" to finish deleting: GET https://rp.j1416320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/11de3a89-d21d-46a0-8fe0-caf4a5581e21 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/11de3a89-d21d-46a0-8fe0-caf4a5581e21", "name": "11de3a89-d21d-46a0-8fe0-caf4a5581e21", "status": "Failed", "startTime": "2026-09-22T11:44:22.052336204Z", "endTime": "2026-09-22T12:08:28.455213082Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t011ljgaaddoqvpihph8n2gttcgf40d still exists (deletion dispatched at 2026-09-22T11:44:23Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001234210>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-7lcrh4lksx7n" to finish deleting: GET https://rp.j1416320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/11de3a89-d21d-46a0-8fe0-caf4a5581e21 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/11de3a89-d21d-46a0-8fe0-caf4a5581e21", "name": "11de3a89-d21d-46a0-8fe0-caf4a5581e21", "status": "Failed", "startTime": "2026-09-22T11:44:22.052336204Z", "endTime": "2026-09-22T12:08:28.455213082Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t011ljgaaddoqvpihph8n2gttcgf40d still exists (deletion dispatched at 2026-09-22T11:44:23Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
09:57:14	2h 11m 31s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 5d48355	centralus	n/a	passed	#7068 (open)	0	n/a
09:57:01	2h 35m 54s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head f8180cd	canadacentral	alert, e2e	failed	#6688 (open)	2	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 09:57:01 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0021de108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-4ztvzt" in resourcegroup="rg-np-version-upgrade-4ztvzt-vtjknvn9vdn5" to finish deleting: GET https://rp.j0107904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2df4b80c-429b-4a34-8f6b-ddc0ca6cee78 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2df4b80c-429b-4a34-8f6b-ddc0ca6cee78", "name": "2df4b80c-429b-4a34-8f6b-ddc0ca6cee78", "status": "Failed", "startTime": "2026-09-22T11:52:44.144339328Z", "endTime": "2026-09-22T12:16:46.994550731Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t016uc314bu49dg4lc99gfogfb8nvr9 still exists (deletion dispatched at 2026-09-22T11:52:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0021de108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-4ztvzt" in resourcegroup="rg-np-version-upgrade-4ztvzt-vtjknvn9vdn5" to finish deleting: GET https://rp.j0107904.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2df4b80c-429b-4a34-8f6b-ddc0ca6cee78 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2df4b80c-429b-4a34-8f6b-ddc0ca6cee78", "name": "2df4b80c-429b-4a34-8f6b-ddc0ca6cee78", "status": "Failed", "startTime": "2026-09-22T11:52:44.144339328Z", "endTime": "2026-09-22T12:16:46.994550731Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t016uc314bu49dg4lc99gfogfb8nvr9 still exists (deletion dispatched at 2026-09-22T11:52:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] ServiceMemoryDrift fired 09:57:01 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T12:00:17Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j0107904-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-vw9hz", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-vw9hz (namespace arobit) on cluster ci01-j0107904-svc has averaged 127.7% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
09:46:41	59m 24s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 8e51fd5 · merge 8e51fd5	canadacentral	provision	failed	★#6439 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code ErrCode_InsufficientVCPUQuota; detail message Insufficient vcpu quota requested <count>, remaining <count> for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: <url> Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.; provider Microsoft.ContainerService 09:46:41 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2) · suite templatize-pipeline Show raw failure time=2026-09-22T10:31:08.961Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n Kind: Shell\n Command: ./aks-cluster-create\n" time=2026-09-22T10:38:47.425Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-22T10:31:08.979907747Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\"}\n{\"time\":\"2026-09-22T10:31:11.06396002Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-22T10:31:11.22661971Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-22T10:38:47.423846978Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"F61EEEEBCFA56868\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2/agentPools/infra1\",\"resourceName\":\"ci01-j4070272-mgmt-2/infra1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T10:36:41.5611028Z\",\"duration\":\"PT1M39.7596603S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T10:38:47.423898879Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"8E07C9049ECB8DEF\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2/agentPools/infra2\",\"resourceName\":\"ci01-j4070272-mgmt-2/infra2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T10:36:41.5611028Z\",\"duration\":\"PT1M39.7909151S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T10:38:47.423911859Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"C8376C88B6FADF35\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2/agentPools/userswft1\",\"resourceName\":\"ci01-j4070272-mgmt-2/userswft1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T10:36:41.5611028Z\",\"duration\":\"PT4.063355S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 18 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T10:38:47.42394617Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"2DFDEDAE29CB0E2C\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2/agentPools/userswft2\",\"resourceName\":\"ci01-j4070272-mgmt-2/userswft2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T10:36:41.5611028Z\",\"duration\":\"PT3.9227798S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 18 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T10:38:47.423968521Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"04012E211BF318FE\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2/agentPools/userswft3\",\"resourceName\":\"ci01-j4070272-mgmt-2/userswft3\",\"resourceType\":\"Microsoft.ContainerService/managedClusters/agentPools\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T10:31:16.3138337Z\",\"duration\":\"PT5M28.8419351S\",\"statusCode\":\"BadRequest\",\"statusMessage\":{\"error\":{\"code\":\"ErrCode_InsufficientVCPUQuota\",\"message\":\"Insufficient vcpu quota requested 80, remaining 18 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T10:38:47.424009742Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"F96D9B5A6875A606\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2\",\"resourceName\":\"ci01-j4070272-mgmt-2\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Succeeded\",\"timestamp\":\"2026-09-22T10:31:16.3138337Z\",\"duration\":\"PT5M25.0918247S\",\"statusCode\":\"OK\",\"statusMessage\":null}\n{\"time\":\"2026-09-22T10:38:47.424030272Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j4070272-mgmt-2\",\"clusterName\":\"ci01-j4070272-mgmt-2\",\"region\":\"canadacentral\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584115338140513340\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"status\\\": \\\"Failed\\\",\\n \\\"error\\\": {\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n \\\"details\\\": [\\n {\\n \\\"code\\\": \\\"BadRequest\\\",\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2/agentPools/userswft1\\\",\\n \\\"message\\\": \\\"{\\\\r\\\\n \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n \\\\\\\"details\\\\\\\": null,\\\\r\\\\n \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 18 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n },\\n {\\n \\\"code\\\": \\\"BadRequest\\\",\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2/agentPools/userswft2\\\",\\n \\\"message\\\": \\\"{\\\\r\\\\n \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n \\\\\\\"details\\\\\\\": null,\\\\r\\\\n \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 18 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n },\\n {\\n \\\"code\\\": \\\"BadRequest\\\",\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4070272-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j4070272-mgmt-2/agentPools/userswft3\\\",\\n \\\"message\\\": \\\"{\\\\r\\\\n \\\\\\\"code\\\\\\\": \\\\\\\"ErrCode_InsufficientVCPUQuota\\\\\\\",\\\\r\\\\n \\\\\\\"details\\\\\\\": null,\\\\r\\\\n \\\\\\\"message\\\\\\\": \\\\\\\"Insufficient vcpu quota requested 80, remaining 18 for family StandardDdsv6Family for region canadacentral. If you want to increase the quota, please follow this instruction: https://learn.microsoft.com/en-us/azure/quotas/view-quotas. Surge nodes would also consume vcpu quota, please consider use smaller maxSurge or use maxUnavailable to proceed upgrade without surge nodes, details: aka.ms/aks/maxUnavailable.\\\\\\\",\\\\r\\\\n \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\n }\\n ]\\n }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"7m38.444136305s\"}\n exit status 1"
09:29:43	2h 2m 7s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 96d978e	westus3	e2e	failed	#7031 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 09:29:43 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00194c2e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-downgrade-wqtjvn" in resourcegroup="rg-np-downgrade-wqtjvn-6zvp2wkzd4qs" to finish deleting: GET https://rp.j5680512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a4570083-37da-40f8-a2e2-fca3e9ade304 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a4570083-37da-40f8-a2e2-fca3e9ade304", "name": "a4570083-37da-40f8-a2e2-fca3e9ade304", "status": "Failed", "startTime": "2026-09-22T10:49:11.895073397Z", "endTime": "2026-09-22T11:13:18.36225794Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t00dlbbm98iflefquha48s4p3gue5g9 still exists (deletion dispatched at 2026-09-22T10:49:12Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00194c2e8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-downgrade-wqtjvn" in resourcegroup="rg-np-downgrade-wqtjvn-6zvp2wkzd4qs" to finish deleting: GET https://rp.j5680512.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a4570083-37da-40f8-a2e2-fca3e9ade304 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a4570083-37da-40f8-a2e2-fca3e9ade304", "name": "a4570083-37da-40f8-a2e2-fca3e9ade304", "status": "Failed", "startTime": "2026-09-22T10:49:11.895073397Z", "endTime": "2026-09-22T11:13:18.36225794Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t00dlbbm98iflefquha48s4p3gue5g9 still exists (deletion dispatched at 2026-09-22T10:49:12Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
09:08:57	2h 8m 38s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 383b384	westus3	alert	failed	#7041 (open)	2	alert (2) alert [svc] FrontendPathMedianLatency fired 09:08:57 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T10:30:51Z Ended: 2026-09-22T10:46:49Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j5854208-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j5854208-svc). alert [svc] ServiceMemoryDrift fired 09:08:57 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T11:02:38Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j5854208-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-vdftx", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-vdftx (namespace arobit) on cluster ci01-j5854208-svc has averaged 126.8% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
09:02:39	2h 20m 52s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head ad72b9c · merge ad72b9c	canadacentral	n/a	passed	#6945 (merged)	0	n/a
09:02:32	2h 18m 26s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 170f768	canadacentral	e2e	failed	#6128 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 09:02:32 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.33 Unexpected error: <fmt.wrapError \| 0xc0013bb900>: failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-v799gb" in resourcegroup="rg-np-version-upgrade-v799gb-zmkrwb8zmjvk" to finish creating: GET https://rp.j5524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/28ed3a88-9cec-48ac-aeee-bbf9bd78e17f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/28ed3a88-9cec-48ac-aeee-bbf9bd78e17f", "name": "28ed3a88-9cec-48ac-aeee-bbf9bd78e17f", "status": "Failed", "startTime": "2026-09-22T10:16:17.959988536Z", "endTime": "2026-09-22T10:35:18.744539107Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.33 Unexpected error: <fmt.wrapError \| 0xc0013bb900>: failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-v799gb" in resourcegroup="rg-np-version-upgrade-v799gb-zmkrwb8zmjvk" to finish creating: GET https://rp.j5524864.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/28ed3a88-9cec-48ac-aeee-bbf9bd78e17f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/28ed3a88-9cec-48ac-aeee-bbf9bd78e17f", "name": "28ed3a88-9cec-48ac-aeee-bbf9bd78e17f", "status": "Failed", "startTime": "2026-09-22T10:16:17.959988536Z", "endTime": "2026-09-22T10:35:18.744539107Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred
09:02:24	2h 41m 53s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head a65a65d	westus3	alert, e2e	failed	#6756 (open)	2	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 09:02:24 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001b460a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-9snnbjlt9gmm" to finish deleting: GET https://rp.j6492928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c058261-e695-445d-9278-032944e20374 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c058261-e695-445d-9278-032944e20374", "name": "6c058261-e695-445d-9278-032944e20374", "status": "Failed", "startTime": "2026-09-22T11:00:25.615396611Z", "endTime": "2026-09-22T11:24:30.827375508Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t00an45ebhqqrvdh3i5r6elagrf94qf still exists (deletion dispatched at 2026-09-22T11:00:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001b460a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-9snnbjlt9gmm" to finish deleting: GET https://rp.j6492928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c058261-e695-445d-9278-032944e20374 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6c058261-e695-445d-9278-032944e20374", "name": "6c058261-e695-445d-9278-032944e20374", "status": "Failed", "startTime": "2026-09-22T11:00:25.615396611Z", "endTime": "2026-09-22T11:24:30.827375508Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t00an45ebhqqrvdh3i5r6elagrf94qf still exists (deletion dispatched at 2026-09-22T11:00:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] ServiceMemoryDrift fired 09:02:24 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T11:09:39Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j6492928-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-7qnw7", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-7qnw7 (namespace arobit) on cluster ci01-j6492928-svc has averaged 126.5% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
09:02:23	2h 35m 3s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 95f6c7a · merge 95f6c7a	canadacentral	alert, e2e	failed	#6948 (merged)	3	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 09:02:23 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001600258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-6qgtplwqgwxc" to finish deleting: GET https://rp.j4607872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7729f3c0-d445-4340-940f-e4e18c7a926c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7729f3c0-d445-4340-940f-e4e18c7a926c", "name": "7729f3c0-d445-4340-940f-e4e18c7a926c", "status": "Failed", "startTime": "2026-09-22T10:52:23.455866421Z", "endTime": "2026-09-22T11:16:32.851282845Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t008v8tdek58eks5u6a3h20d448qg5d still exists (deletion dispatched at 2026-09-22T10:52:24Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001600258>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-6qgtplwqgwxc" to finish deleting: GET https://rp.j4607872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7729f3c0-d445-4340-940f-e4e18c7a926c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7729f3c0-d445-4340-940f-e4e18c7a926c", "name": "7729f3c0-d445-4340-940f-e4e18c7a926c", "status": "Failed", "startTime": "2026-09-22T10:52:23.455866421Z", "endTime": "2026-09-22T11:16:32.851282845Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t008v8tdek58eks5u6a3h20d448qg5d still exists (deletion dispatched at 2026-09-22T10:52:24Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [svc] FrontendPathLatency fired 09:02:23 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T10:47:24Z Ended: 2026-09-22T11:06:27Z Severity: Sev3 Labels: alertname="FrontendPathLatency", cluster="ci01-j4607872-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning" Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j4607872-svc). alert [svc] FrontendPathMedianLatency fired 09:02:23 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T10:40:24Z Ended: 2026-09-22T10:46:25Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j4607872-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j4607872-svc).
09:02:16	2h 23m 20s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 54049e5 · merge 54049e5	centralus	alert, e2e	failed	#6995 (merged)	3	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 09:02:16 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a22108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-xvl2sk9668pf" to finish deleting: GET https://rp.j1604736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/092084d5-d703-4f92-86f4-eb6914a0f751 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/092084d5-d703-4f92-86f4-eb6914a0f751", "name": "092084d5-d703-4f92-86f4-eb6914a0f751", "status": "Failed", "startTime": "2026-09-22T10:43:39.247737464Z", "endTime": "2026-09-22T11:07:45.926705915Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0064f6bcmvlot3qp483fs9rj85leb0 still exists (deletion dispatched at 2026-09-22T10:43:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a22108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-xvl2sk9668pf" to finish deleting: GET https://rp.j1604736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/092084d5-d703-4f92-86f4-eb6914a0f751 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/092084d5-d703-4f92-86f4-eb6914a0f751", "name": "092084d5-d703-4f92-86f4-eb6914a0f751", "status": "Failed", "startTime": "2026-09-22T10:43:39.247737464Z", "endTime": "2026-09-22T11:07:45.926705915Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2t0064f6bcmvlot3qp483fs9rj85leb0 still exists (deletion dispatched at 2026-09-22T10:43:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [svc] ServiceMemoryDrift fired 09:02:16 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T11:08:02Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j1604736-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-b55rr", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-b55rr (namespace arobit) on cluster ci01-j1604736-svc has averaged 125.8% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml. alert [svc] FrontendPathMedianLatency fired 09:02:16 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T10:32:45Z Ended: 2026-09-22T10:38:46Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j1604736-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1604736-svc).
08:58:41	2h 17m 7s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 571559a · merge 571559a	westus3	alert, e2e	failed	#6516 (merged)	4	e2e (1) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift 08:58:41 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc0010ac9c0>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-smq89jgn4gq5" to finish creating: GET https://rp.j9233280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/37dd9b79-8013-4222-b75a-587194eb2514 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/37dd9b79-8013-4222-b75a-587194eb2514", "name": "37dd9b79-8013-4222-b75a-587194eb2514", "status": "Failed", "startTime": "2026-09-22T10:03:21.992748868Z", "endTime": "2026-09-22T10:32:22.669673154Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc0010ac9c0>: failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-smq89jgn4gq5" to finish creating: GET https://rp.j9233280.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/37dd9b79-8013-4222-b75a-587194eb2514 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/37dd9b79-8013-4222-b75a-587194eb2514", "name": "37dd9b79-8013-4222-b75a-587194eb2514", "status": "Failed", "startTime": "2026-09-22T10:03:21.992748868Z", "endTime": "2026-09-22T10:32:22.669673154Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred alert (3) alert [svc] FrontendPathLatency fired 08:58:41 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T10:44:39Z Ended: 2026-09-22T10:56:41Z Severity: Sev3 Labels: alertname="FrontendPathLatency", cluster="ci01-j9233280-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j9233280-svc). alert [hcp] KubeStatefulSetReplicasMismatch fired 08:58:41 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-09-22T10:24:38Z Ended: 2026-09-22T10:53:38Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9233280-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9233280/providers/microsoft.monitor/accounts/hcps-j9233280", namespace="ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901", pod="arohcp-monitor-kube-state-metrics-7955558448-q8nsw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901/etcd has not matched the expected number of replicas for longer than 15 minutes. Firing 2: State: Resolved Started: 2026-09-22T10:24:38Z Ended: 2026-09-22T10:50:38Z Severity: Sev3 Labels: alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j9233280-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9233280/providers/microsoft.monitor/accounts/hcps-j9233280", namespace="ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901", pod="arohcp-monitor-kube-state-metrics-7955558448-q8nsw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd" Description: StatefulSet ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901/etcd has not matched the expected number of replicas for longer than 15 minutes. alert [hcp] KubeDeploymentReplicasMismatch fired 08:58:41 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests Show raw failure alert fired 2 time(s) Firing 1: State: Resolved Started: 2026-09-22T10:39:33Z Ended: 2026-09-22T10:52:34Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9233280-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9233280/providers/microsoft.monitor/accounts/hcps-j9233280", namespace="ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901", pod="arohcp-monitor-kube-state-metrics-7955558448-q8nsw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901/capi-provider has not matched the expected number of replicas for longer than 30 minutes. Firing 2: State: Resolved Started: 2026-09-22T10:49:32Z Ended: 2026-09-22T10:56:35Z Severity: Sev3 Labels: alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j9233280-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.199:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9233280/providers/microsoft.monitor/accounts/hcps-j9233280", namespace="ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901", pod="arohcp-monitor-kube-state-metrics-7955558448-q8nsw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning" Description: Deployment ocm-arohcpci01-2t008r4997p9pdhbin6riaqnkh7p7od9-v20260901/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
08:48:41	2h 3m 26s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 02f5585 · merge 02f5585	canadacentral	n/a	passed	#5724 (merged)	0	n/a
08:36:13	12m 7s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head a65a65d	n/a	unknown	failed	#6756 (open)	1	non-artifact-backed failure (no junit artifacts)
08:00:53	2h 16m 33s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 87e40a0	westus3	alert, e2e	failed	#6922 (open)	2	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 08:00:53 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a24150>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-8cpg6p" in resourcegroup="rg-cp-ystream-upgrade-4-22-9jjn6d27plq8" to finish deleting: GET https://rp.j1352576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba57e58c-f8d4-4d4b-904a-f3e851086567 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba57e58c-f8d4-4d4b-904a-f3e851086567", "name": "ba57e58c-f8d4-4d4b-904a-f3e851086567", "status": "Failed", "startTime": "2026-09-22T09:32:28.672968194Z", "endTime": "2026-09-22T09:56:36.170444319Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svvbd6lttd70rvgqctu642sbsj2vlqt still exists (deletion dispatched at 2026-09-22T09:32:48Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a24150>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-8cpg6p" in resourcegroup="rg-cp-ystream-upgrade-4-22-9jjn6d27plq8" to finish deleting: GET https://rp.j1352576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba57e58c-f8d4-4d4b-904a-f3e851086567 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/ba57e58c-f8d4-4d4b-904a-f3e851086567", "name": "ba57e58c-f8d4-4d4b-904a-f3e851086567", "status": "Failed", "startTime": "2026-09-22T09:32:28.672968194Z", "endTime": "2026-09-22T09:56:36.170444319Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svvbd6lttd70rvgqctu642sbsj2vlqt still exists (deletion dispatched at 2026-09-22T09:32:48Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] FrontendPathMedianLatency fired 08:00:53 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T09:16:29Z Ended: 2026-09-22T09:43:29Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j1352576-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1352576-svc).
07:14:27	2h 4m 55s	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 6c5cfd0 · merge a0bf660	westus3	alert, e2e	failed	#6503 (merged)	3	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 07:14:27 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00295c090>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-k8jcmpkl6wgx" to finish deleting: GET https://rp.j9742592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2ccd7ef7-ba19-4730-b390-c22a5cb16ed8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2ccd7ef7-ba19-4730-b390-c22a5cb16ed8", "name": "2ccd7ef7-ba19-4730-b390-c22a5cb16ed8", "status": "Failed", "startTime": "2026-09-22T08:30:31.706415617Z", "endTime": "2026-09-22T08:54:37.036482039Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svughrdal726o55rr99lgebhg86q31q still exists (deletion dispatched at 2026-09-22T08:30:33Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00295c090>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="agg-cluster" in resourcegroup="feature-aggregation-k8jcmpkl6wgx" to finish deleting: GET https://rp.j9742592.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2ccd7ef7-ba19-4730-b390-c22a5cb16ed8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2ccd7ef7-ba19-4730-b390-c22a5cb16ed8", "name": "2ccd7ef7-ba19-4730-b390-c22a5cb16ed8", "status": "Failed", "startTime": "2026-09-22T08:30:31.706415617Z", "endTime": "2026-09-22T08:54:37.036482039Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svughrdal726o55rr99lgebhg86q31q still exists (deletion dispatched at 2026-09-22T08:30:33Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [svc] FrontendPathLatency fired 07:14:27 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T08:40:36Z Ended: 2026-09-22T08:56:35Z Severity: Sev3 Labels: alertname="FrontendPathLatency", cluster="ci01-j9742592-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters", severity="warning" Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters exceeded 1 second over the past 30 minutes (cluster ci01-j9742592-svc). alert [svc] BackendControllerQueueDepthHigh fired 07:14:27 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T08:16:37Z Ended: 2026-09-22T08:34:37Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9742592-svc", component="backend", name="clusterresources", region="westus3", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
07:09:13	2h 21m 55s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 7cf0a28	centralus	e2e	failed	#6364 (open)	4	e2e (4) DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: no such host): context deadline exceeded 07:09:13 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:115]: failed to verify simple web app runs on cluster "cluster-hs-pre-84mf8p" Unexpected error: <fmt.wrapError \| 0xc001186040>: DNS for route host agnhost-e2e-sample-app-zjd8b.apps.aro.g7i1i4o8r4c6r5x.5s62.j9281536.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-zjd8b.apps.aro.g7i1i4o8r4c6r5x.5s62.j9281536.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-zjd8b.apps.aro.g7i1i4o8r4c6r5x.5s62.j9281536.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:115]: failed to verify simple web app runs on cluster "cluster-hs-pre-84mf8p" Unexpected error: <fmt.wrapError \| 0xc001186040>: DNS for route host agnhost-e2e-sample-app-zjd8b.apps.aro.g7i1i4o8r4c6r5x.5s62.j9281536.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-zjd8b.apps.aro.g7i1i4o8r4c6r5x.5s62.j9281536.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-zjd8b.apps.aro.g7i1i4o8r4c6r5x.5s62.j9281536.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 07:09:13 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error: <errors.joinError \| 0xc0012181b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-7dpf6w54wkcq" to finish deleting: GET https://rp.j9281536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b5834841-df9c-4967-9c2d-7ef56f8c94f8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b5834841-df9c-4967-9c2d-7ef56f8c94f8", "name": "b5834841-df9c-4967-9c2d-7ef56f8c94f8", "status": "Failed", "startTime": "2026-09-22T08:48:33.543198293Z", "endTime": "2026-09-22T09:12:40.623611763Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svudbr2hssobmqj9amh0pf72td9kpdk still exists (deletion dispatched at 2026-09-22T08:48:35Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error: <errors.joinError \| 0xc0012181b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-7dpf6w54wkcq" to finish deleting: GET https://rp.j9281536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b5834841-df9c-4967-9c2d-7ef56f8c94f8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b5834841-df9c-4967-9c2d-7ef56f8c94f8", "name": "b5834841-df9c-4967-9c2d-7ef56f8c94f8", "status": "Failed", "startTime": "2026-09-22T08:48:33.543198293Z", "endTime": "2026-09-22T09:12:40.623611763Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svudbr2hssobmqj9amh0pf72td9kpdk still exists (deletion dispatched at 2026-09-22T08:48:35Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 07:09:13 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error: <errors.joinError \| 0xc001bd41e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-xlzps9" in resourcegroup="rg-np-version-upgrade-xlzps9-5jp45ggx94q5" to finish deleting: GET https://rp.j9281536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bab02b80-e2c9-493c-8702-48bdacd5cfbd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bab02b80-e2c9-493c-8702-48bdacd5cfbd", "name": "bab02b80-e2c9-493c-8702-48bdacd5cfbd", "status": "Failed", "startTime": "2026-09-22T08:41:40.162628083Z", "endTime": "2026-09-22T09:05:40.589950549Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svudcl0k8f6n7hat0l3mq2kc5oo9gbd still exists (deletion dispatched at 2026-09-22T08:41:42Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error: <errors.joinError \| 0xc001bd41e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-xlzps9" in resourcegroup="rg-np-version-upgrade-xlzps9-5jp45ggx94q5" to finish deleting: GET https://rp.j9281536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bab02b80-e2c9-493c-8702-48bdacd5cfbd -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bab02b80-e2c9-493c-8702-48bdacd5cfbd", "name": "bab02b80-e2c9-493c-8702-48bdacd5cfbd", "status": "Failed", "startTime": "2026-09-22T08:41:40.162628083Z", "endTime": "2026-09-22T09:05:40.589950549Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svudcl0k8f6n7hat0l3mq2kc5oo9gbd still exists (deletion dispatched at 2026-09-22T08:41:42Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 07:09:13 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error: <errors.joinError \| 0xc0015100c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-9sdm78" in resourcegroup="rg-cp-ystream-upgrade-5-0-sb4lq8xcf9k6" to finish deleting: GET https://rp.j9281536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a7133fbd-4d17-4f3b-aa59-ab3b1ce58d8a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a7133fbd-4d17-4f3b-aa59-ab3b1ce58d8a", "name": "a7133fbd-4d17-4f3b-aa59-ab3b1ce58d8a", "status": "Failed", "startTime": "2026-09-22T08:30:59.74305005Z", "endTime": "2026-09-22T08:55:00.567673289Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svudf4qkt2oamjltcf461o8t5os6if5 still exists (deletion dispatched at 2026-09-22T08:31:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:289]: Unexpected error: <errors.joinError \| 0xc0015100c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-5-0-9sdm78" in resourcegroup="rg-cp-ystream-upgrade-5-0-sb4lq8xcf9k6" to finish deleting: GET https://rp.j9281536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a7133fbd-4d17-4f3b-aa59-ab3b1ce58d8a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a7133fbd-4d17-4f3b-aa59-ab3b1ce58d8a", "name": "a7133fbd-4d17-4f3b-aa59-ab3b1ce58d8a", "status": "Failed", "startTime": "2026-09-22T08:30:59.74305005Z", "endTime": "2026-09-22T08:55:00.567673289Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svudf4qkt2oamjltcf461o8t5os6if5 still exists (deletion dispatched at 2026-09-22T08:31:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
07:01:28	1h 8m 43s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 4561955	canadacentral	provision	failed	⚠#6869 (open)	1	provision (1) error running Shell Step, failed to execute shell command: {"level":"INFO","msg":"starting cluster reconciliation","component":"aks-cluster-create","subscriptionID":"<redacted-id>","resourceGroup":"hcp-underlay-ci01-j9153920-mgmt-2","clusterName":"ci01-j915392 07:01:28 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=2) · suite templatize-pipeline Show raw failure time=2026-09-22T07:33:19.014Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n Kind: Shell\n Command: ./aks-cluster-create\n" time=2026-09-22T08:03:18.587Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-22T07:33:19.023908785Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9153920-mgmt-2\",\"clusterName\":\"ci01-j9153920-mgmt-2\",\"region\":\"canadacentral\"}\n{\"time\":\"2026-09-22T07:33:21.03768985Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9153920-mgmt-2\",\"clusterName\":\"ci01-j9153920-mgmt-2\",\"region\":\"canadacentral\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-22T07:33:21.167202045Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j9153920-mgmt-2\",\"clusterName\":\"ci01-j9153920-mgmt-2\",\"region\":\"canadacentral\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n signal: killed"
06:52:43	1h 7m 8s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head b80303c	centralus	provision	failed	#7060 (closed)	1	provision (1) failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded 06:52:43 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite templatize-pipeline Show raw failure time=2026-09-22T07:21:38.848Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-09-22T07:21:40.354Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-09-22T07:21:41.548Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=d03eff9eca8ce6668b3735feb0fed36048a18f0177222fc337f65cc4e325d887 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4723968-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fd03eff9eca8ce6668b3735feb0fed36048a18f0177222fc337f65cc4e325d887 time=2026-09-22T07:51:38.563Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: context deadline exceeded"
06:46:25	2h 13m 42s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 170f768	westus3	e2e	failed	#6128 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 06:46:25 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001857b30>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-9bf8n6" in resourcegroup="rg-zstream-upgrade-5-0-nkz6rtcklccm" to finish deleting: GET https://rp.j5510144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fad47a2e-73a1-4972-a714-79b849fba16a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fad47a2e-73a1-4972-a714-79b849fba16a", "name": "fad47a2e-73a1-4972-a714-79b849fba16a", "status": "Failed", "startTime": "2026-09-22T08:11:14.087337109Z", "endTime": "2026-09-22T08:35:22.583136314Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svu5bos2fu9mlb7h90mr26252ojhi8d still exists (deletion dispatched at 2026-09-22T08:11:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001857b30>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-5-0-9bf8n6" in resourcegroup="rg-zstream-upgrade-5-0-nkz6rtcklccm" to finish deleting: GET https://rp.j5510144.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fad47a2e-73a1-4972-a714-79b849fba16a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/fad47a2e-73a1-4972-a714-79b849fba16a", "name": "fad47a2e-73a1-4972-a714-79b849fba16a", "status": "Failed", "startTime": "2026-09-22T08:11:14.087337109Z", "endTime": "2026-09-22T08:35:22.583136314Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svu5bos2fu9mlb7h90mr26252ojhi8d still exists (deletion dispatched at 2026-09-22T08:11:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
06:46:14	2h 11m 45s	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head eca3647 · merge 23179a5	centralus	alert, e2e	failed	#7066 (merged)	3	e2e (2) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 06:46:14 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000848ab0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-2hzskj2pr2m2" to finish deleting: GET https://rp.j5704064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b8310f47-3419-4b91-8a4e-4790ddca0cbb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b8310f47-3419-4b91-8a4e-4790ddca0cbb", "name": "b8310f47-3419-4b91-8a4e-4790ddca0cbb", "status": "Failed", "startTime": "2026-09-22T08:12:16.220105938Z", "endTime": "2026-09-22T08:36:24.406787177Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svu302f6dr3da93kcsml9jev780gduh still exists (deletion dispatched at 2026-09-22T08:12:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000848ab0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-2hzskj2pr2m2" to finish deleting: GET https://rp.j5704064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b8310f47-3419-4b91-8a4e-4790ddca0cbb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b8310f47-3419-4b91-8a4e-4790ddca0cbb", "name": "b8310f47-3419-4b91-8a4e-4790ddca0cbb", "status": "Failed", "startTime": "2026-09-22T08:12:16.220105938Z", "endTime": "2026-09-22T08:36:24.406787177Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svu302f6dr3da93kcsml9jev780gduh still exists (deletion dispatched at 2026-09-22T08:12:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 06:46:14 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000a665d0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-5ckwfmkg58dx" to finish deleting: GET https://rp.j5704064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9c4a5f58-2281-43f4-9356-11220cd3316d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9c4a5f58-2281-43f4-9356-11220cd3316d", "name": "9c4a5f58-2281-43f4-9356-11220cd3316d", "status": "Failed", "startTime": "2026-09-22T08:01:34.997609657Z", "endTime": "2026-09-22T08:25:34.356075446Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svu3n2uvu3jjl757qfdq4ivet0ki4e5 still exists (deletion dispatched at 2026-09-22T08:01:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000a665d0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cilium-cluster" in resourcegroup="complex-cilium-kv-5ckwfmkg58dx" to finish deleting: GET https://rp.j5704064.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9c4a5f58-2281-43f4-9356-11220cd3316d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9c4a5f58-2281-43f4-9356-11220cd3316d", "name": "9c4a5f58-2281-43f4-9356-11220cd3316d", "status": "Failed", "startTime": "2026-09-22T08:01:34.997609657Z", "endTime": "2026-09-22T08:25:34.356075446Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svu3n2uvu3jjl757qfdq4ivet0ki4e5 still exists (deletion dispatched at 2026-09-22T08:01:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] FrontendPathMedianLatency fired 06:46:14 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T08:02:38Z Ended: 2026-09-22T08:15:39Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j5704064-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j5704064-svc).
06:37:44	2h 7m 8s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 54049e5 · merge 54049e5	canadacentral	e2e	failed	#6995 (merged)	3	e2e (3) ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 06:37:44 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.38 Unexpected error: <fmt.wrapError \| 0xc0011159a0>: failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-dxqcht" in resourcegroup="rg-np-noedge-dxqcht-6zchcrtc2jzz" to finish creating: GET https://rp.j5260288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9158be66-cd26-49d4-a60f-90293011d19d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9158be66-cd26-49d4-a60f-90293011d19d", "name": "9158be66-cd26-49d4-a60f-90293011d19d", "status": "Failed", "startTime": "2026-09-22T07:34:55.729794931Z", "endTime": "2026-09-22T07:53:56.208999403Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:319]: failed to create nodepool npnoedge-4-20 at version 4.20.38 Unexpected error: <fmt.wrapError \| 0xc0011159a0>: failed to create NodePool npnoedge-4-20: failed waiting for nodepool="npnoedge-4-20" for cluster "np-noedge-dxqcht" in resourcegroup="rg-np-noedge-dxqcht-6zchcrtc2jzz" to finish creating: GET https://rp.j5260288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9158be66-cd26-49d4-a60f-90293011d19d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9158be66-cd26-49d4-a60f-90293011d19d", "name": "9158be66-cd26-49d4-a60f-90293011d19d", "status": "Failed", "startTime": "2026-09-22T07:34:55.729794931Z", "endTime": "2026-09-22T07:53:56.208999403Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 06:37:44 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0008fe348>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-9c9tl7wq8vk4" to finish deleting: GET https://rp.j5260288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fc7ae76f-bb09-4015-9244-5d0a185fd088 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fc7ae76f-bb09-4015-9244-5d0a185fd088", "name": "fc7ae76f-bb09-4015-9244-5d0a185fd088", "status": "Failed", "startTime": "2026-09-22T07:58:38.947860134Z", "endTime": "2026-09-22T08:22:41.917812685Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svttdmiuctqkhavuc9550dhp1jpput2 still exists (deletion dispatched at 2026-09-22T07:58:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0008fe348>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-9c9tl7wq8vk4" to finish deleting: GET https://rp.j5260288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fc7ae76f-bb09-4015-9244-5d0a185fd088 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/fc7ae76f-bb09-4015-9244-5d0a185fd088", "name": "fc7ae76f-bb09-4015-9244-5d0a185fd088", "status": "Failed", "startTime": "2026-09-22T07:58:38.947860134Z", "endTime": "2026-09-22T08:22:41.917812685Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svttdmiuctqkhavuc9550dhp1jpput2 still exists (deletion dispatched at 2026-09-22T07:58:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred -------------------------------------------------------------------------------- 06:37:44 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-hb6mtzv8hm6q" to finish creating: GET https://rp.j5260288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b4e55441-95a5-466c-adac-c574d8bcdb81 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b4e55441-95a5-466c-adac-c574d8bcdb81", "name": "b4e55441-95a5-466c-adac-c574d8bcdb81", "status": "Failed", "startTime": "2026-09-22T07:34:05.103384159Z", "endTime": "2026-09-22T07:53:06.353871266Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ] Expected <[]error \| len:1, cap:1>: ... to be empty fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-hb6mtzv8hm6q" to finish creating: GET https://rp.j5260288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b4e55441-95a5-466c-adac-c574d8bcdb81 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b4e55441-95a5-466c-adac-c574d8bcdb81", "name": "b4e55441-95a5-466c-adac-c574d8bcdb81", "status": "Failed", "startTime": "2026-09-22T07:34:05.103384159Z", "endTime": "2026-09-22T07:53:06.353871266Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ] Expected <[]error \| len:1, cap:1>: ... to be empty
06:28:15	2h 14m 30s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 383b384	canadacentral	alert, e2e	failed	#7041 (open)	4	e2e (3) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 06:28:15 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0010d8438>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-dg-minor-gzv72w" in resourcegroup="rg-np-dg-minor-gzv72w-qbl26k5f9hc6" to finish deleting: GET https://rp.j6116608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/70b7b6e5-3879-4a3d-8927-7d4f3c24a792 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/70b7b6e5-3879-4a3d-8927-7d4f3c24a792", "name": "70b7b6e5-3879-4a3d-8927-7d4f3c24a792", "status": "Failed", "startTime": "2026-09-22T08:00:43.888264873Z", "endTime": "2026-09-22T08:24:52.37976402Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svtqs2972vd4pkj0oea8s7g10qqc55p still exists (deletion dispatched at 2026-09-22T08:00:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0010d8438>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-dg-minor-gzv72w" in resourcegroup="rg-np-dg-minor-gzv72w-qbl26k5f9hc6" to finish deleting: GET https://rp.j6116608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/70b7b6e5-3879-4a3d-8927-7d4f3c24a792 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/70b7b6e5-3879-4a3d-8927-7d4f3c24a792", "name": "70b7b6e5-3879-4a3d-8927-7d4f3c24a792", "status": "Failed", "startTime": "2026-09-22T08:00:43.888264873Z", "endTime": "2026-09-22T08:24:52.37976402Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svtqs2972vd4pkj0oea8s7g10qqc55p still exists (deletion dispatched at 2026-09-22T08:00:45Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 06:28:15 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001b00108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-m5blp6" in resourcegroup="admin-credential-lifecycle-test-cdw82gf9mgkj" to finish deleting: GET https://rp.j6116608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34fd4832-1a9c-4c2b-8b21-92a0603c3364 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34fd4832-1a9c-4c2b-8b21-92a0603c3364", "name": "34fd4832-1a9c-4c2b-8b21-92a0603c3364", "status": "Failed", "startTime": "2026-09-22T07:41:21.159244981Z", "endTime": "2026-09-22T08:05:24.861502692Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svtqqq0lva06van6d9rveis55f6keot still exists (deletion dispatched at 2026-09-22T07:41:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001b00108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="admin-cred-lifecycle-m5blp6" in resourcegroup="admin-credential-lifecycle-test-cdw82gf9mgkj" to finish deleting: GET https://rp.j6116608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34fd4832-1a9c-4c2b-8b21-92a0603c3364 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/34fd4832-1a9c-4c2b-8b21-92a0603c3364", "name": "34fd4832-1a9c-4c2b-8b21-92a0603c3364", "status": "Failed", "startTime": "2026-09-22T07:41:21.159244981Z", "endTime": "2026-09-22T08:05:24.861502692Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svtqqq0lva06van6d9rveis55f6keot still exists (deletion dispatched at 2026-09-22T07:41:22Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 06:28:15 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl" Unexpected error: <errors.joinError \| 0xc001349428>: failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-4zk8nv92zkwr" to finish creating: GET https://rp.j6116608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/325e6a8c-0d6c-4807-996c-1351e4e5c6e0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/325e6a8c-0d6c-4807-996c-1351e4e5c6e0", "name": "325e6a8c-0d6c-4807-996c-1351e4e5c6e0", "status": "Failed", "startTime": "2026-09-22T07:27:52.829680188Z", "endTime": "2026-09-22T07:46:55.075402091Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl" Unexpected error: <errors.joinError \| 0xc001349428>: failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-4zk8nv92zkwr" to finish creating: GET https://rp.j6116608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/325e6a8c-0d6c-4807-996c-1351e4e5c6e0 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/325e6a8c-0d6c-4807-996c-1351e4e5c6e0", "name": "325e6a8c-0d6c-4807-996c-1351e4e5c6e0", "status": "Failed", "startTime": "2026-09-22T07:27:52.829680188Z", "endTime": "2026-09-22T07:46:55.075402091Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] ServiceMemoryDrift fired 06:28:15 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T08:31:25Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j6116608-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-zm7fl", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-zm7fl (namespace arobit) on cluster ci01-j6116608-svc has averaged 126.1% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
06:00:11	2h 14m 25s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 1c995b8	centralus	e2e	failed	#7053 (open)	3	e2e (3) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 06:00:11 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000a580f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-7hhc2zr8vjxs" to finish deleting: GET https://rp.j5008384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a3241016-1253-4da6-8623-c49473e08ffb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a3241016-1253-4da6-8623-c49473e08ffb", "name": "a3241016-1253-4da6-8623-c49473e08ffb", "status": "Failed", "startTime": "2026-09-22T07:21:14.063772758Z", "endTime": "2026-09-22T07:45:20.292071261Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svtbp449eoj3qf5nbs7ctd9n6sit8tg still exists (deletion dispatched at 2026-09-22T07:21:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000a580f0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-7hhc2zr8vjxs" to finish deleting: GET https://rp.j5008384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a3241016-1253-4da6-8623-c49473e08ffb -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a3241016-1253-4da6-8623-c49473e08ffb", "name": "a3241016-1253-4da6-8623-c49473e08ffb", "status": "Failed", "startTime": "2026-09-22T07:21:14.063772758Z", "endTime": "2026-09-22T07:45:20.292071261Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svtbp449eoj3qf5nbs7ctd9n6sit8tg still exists (deletion dispatched at 2026-09-22T07:21:15Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded 06:00:11 UTC · failed at e2e · test ARO-HCP HyperShift Presubmit should create a cluster and nodepool to completion · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:115]: failed to verify simple web app runs on cluster "cluster-hs-pre-svhj5w" Unexpected error: <fmt.wrapError \| 0xc0017e4720>: DNS for route host agnhost-e2e-sample-app-sxj6m.apps.aro.b5e0p3v1r0t3r4t.l2s4.j5008384.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-sxj6m.apps.aro.b5e0p3v1r0t3r4t.l2s4.j5008384.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sxj6m.apps.aro.b5e0p3v1r0t3r4t.l2s4.j5008384.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:115]: failed to verify simple web app runs on cluster "cluster-hs-pre-svhj5w" Unexpected error: <fmt.wrapError \| 0xc0017e4720>: DNS for route host agnhost-e2e-sample-app-sxj6m.apps.aro.b5e0p3v1r0t3r4t.l2s4.j5008384.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-sxj6m.apps.aro.b5e0p3v1r0t3r4t.l2s4.j5008384.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sxj6m.apps.aro.b5e0p3v1r0t3r4t.l2s4.j5008384.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 06:00:11 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001bac198>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-2c7gjb8qcjb8" to finish deleting: GET https://rp.j5008384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8a37ffb4-e28e-4493-a3e8-361ee859d5a8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8a37ffb4-e28e-4493-a3e8-361ee859d5a8", "name": "8a37ffb4-e28e-4493-a3e8-361ee859d5a8", "status": "Failed", "startTime": "2026-09-22T07:31:49.427817516Z", "endTime": "2026-09-22T07:55:50.330781697Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svtbhu4l0og0la7ciu7nisn164rc9se still exists (deletion dispatched at 2026-09-22T07:31:51Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001bac198>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-2c7gjb8qcjb8" to finish deleting: GET https://rp.j5008384.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8a37ffb4-e28e-4493-a3e8-361ee859d5a8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8a37ffb4-e28e-4493-a3e8-361ee859d5a8", "name": "8a37ffb4-e28e-4493-a3e8-361ee859d5a8", "status": "Failed", "startTime": "2026-09-22T07:31:49.427817516Z", "endTime": "2026-09-22T07:55:50.330781697Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svtbhu4l0og0la7ciu7nisn164rc9se still exists (deletion dispatched at 2026-09-22T07:31:51Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
05:59:59	2h 12m 27s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head e8df860 · merge e8df860	canadacentral	e2e	failed	#5846 (merged)	2	e2e (2) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 05:59:59 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0019a42b8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-prv579" in resourcegroup="rg-candidate-4-20-s4b6jx4lj9fq" to finish deleting: GET https://rp.j9926784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a55d2b2a-4b2b-4856-9356-be804c0b554f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a55d2b2a-4b2b-4856-9356-be804c0b554f", "name": "a55d2b2a-4b2b-4856-9356-be804c0b554f", "status": "Failed", "startTime": "2026-09-22T07:21:32.842959151Z", "endTime": "2026-09-22T07:45:33.914148356Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svtfurgd77dtrpps49catdbplg5jon7 still exists (deletion dispatched at 2026-09-22T07:21:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0019a42b8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-candidate-4-20-prv579" in resourcegroup="rg-candidate-4-20-s4b6jx4lj9fq" to finish deleting: GET https://rp.j9926784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a55d2b2a-4b2b-4856-9356-be804c0b554f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a55d2b2a-4b2b-4856-9356-be804c0b554f", "name": "a55d2b2a-4b2b-4856-9356-be804c0b554f", "status": "Failed", "startTime": "2026-09-22T07:21:32.842959151Z", "endTime": "2026-09-22T07:45:33.914148356Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svtfurgd77dtrpps49catdbplg5jon7 still exists (deletion dispatched at 2026-09-22T07:21:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 05:59:59 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0011c8090>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-4vwc4ffksj6w" to finish deleting: GET https://rp.j9926784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1ab4f584-13dc-4529-b3a7-52bf12ff4323 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1ab4f584-13dc-4529-b3a7-52bf12ff4323", "name": "1ab4f584-13dc-4529-b3a7-52bf12ff4323", "status": "Failed", "startTime": "2026-09-22T07:23:42.071372233Z", "endTime": "2026-09-22T07:47:43.944680192Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svth6c8pterthepvbj92trdph5hdiit still exists (deletion dispatched at 2026-09-22T07:23:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0011c8090>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-4vwc4ffksj6w" to finish deleting: GET https://rp.j9926784.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1ab4f584-13dc-4529-b3a7-52bf12ff4323 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1ab4f584-13dc-4529-b3a7-52bf12ff4323", "name": "1ab4f584-13dc-4529-b3a7-52bf12ff4323", "status": "Failed", "startTime": "2026-09-22T07:23:42.071372233Z", "endTime": "2026-09-22T07:47:43.944680192Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svth6c8pterthepvbj92trdph5hdiit still exists (deletion dispatched at 2026-09-22T07:23:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
05:59:56	2h 9m 59s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 95f6c7a · merge 95f6c7a	centralus	alert, e2e	failed	#6948 (merged)	5	e2e (3) DNS for route host <route-host> did not resolve: DNS for <route-host> did not resolve within <duration> (last error: lookup <host> on <dns-server>: dial udp <ip>: i/o timeout): context deadline exceeded 05:59:56 UTC · failed at e2e · test Customer should be able to retrieve cluster and nodepool status active versions · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:157]: failed to verify simple web app runs on v20261001preview cluster "v20261001" Unexpected error: <fmt.wrapError \| 0xc001120bc0>: DNS for route host agnhost-e2e-sample-app-g6m95.apps.aro.v20261001.kkqb.j8719872.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-g6m95.apps.aro.v20261001.kkqb.j8719872.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g6m95.apps.aro.v20261001.kkqb.j8719872.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:157]: failed to verify simple web app runs on v20261001preview cluster "v20261001" Unexpected error: <fmt.wrapError \| 0xc001120bc0>: DNS for route host agnhost-e2e-sample-app-g6m95.apps.aro.v20261001.kkqb.j8719872.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-g6m95.apps.aro.v20261001.kkqb.j8719872.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-g6m95.apps.aro.v20261001.kkqb.j8719872.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 05:59:56 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0016001c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-pbw5cbhdkkv4" to finish deleting: GET https://rp.j8719872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3339f910-8502-48bd-b40c-fd8c2ee3426d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3339f910-8502-48bd-b40c-fd8c2ee3426d", "name": "3339f910-8502-48bd-b40c-fd8c2ee3426d", "status": "Failed", "startTime": "2026-09-22T07:11:19.031336559Z", "endTime": "2026-09-22T07:35:24.846401043Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svtacm4udbvukgr25lmrgtehhtlrsak still exists (deletion dispatched at 2026-09-22T07:11:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0016001c8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ingress-cert-le" in resourcegroup="ingress-cert-le-pbw5cbhdkkv4" to finish deleting: GET https://rp.j8719872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3339f910-8502-48bd-b40c-fd8c2ee3426d -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3339f910-8502-48bd-b40c-fd8c2ee3426d", "name": "3339f910-8502-48bd-b40c-fd8c2ee3426d", "status": "Failed", "startTime": "2026-09-22T07:11:19.031336559Z", "endTime": "2026-09-22T07:35:24.846401043Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svtacm4udbvukgr25lmrgtehhtlrsak still exists (deletion dispatched at 2026-09-22T07:11:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 05:59:56 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0015800a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-fp4cw7tcx7zs" to finish deleting: GET https://rp.j8719872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ef4b3069-c7ef-4624-b3fa-fd8d3ce178d6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ef4b3069-c7ef-4624-b3fa-fd8d3ce178d6", "name": "ef4b3069-c7ef-4624-b3fa-fd8d3ce178d6", "status": "Failed", "startTime": "2026-09-22T07:28:38.432300499Z", "endTime": "2026-09-22T07:52:38.299308463Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svtah7m0vc8lt85cerhp2ucmjjj31gs still exists (deletion dispatched at 2026-09-22T07:28:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0015800a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-fp4cw7tcx7zs" to finish deleting: GET https://rp.j8719872.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ef4b3069-c7ef-4624-b3fa-fd8d3ce178d6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ef4b3069-c7ef-4624-b3fa-fd8d3ce178d6", "name": "ef4b3069-c7ef-4624-b3fa-fd8d3ce178d6", "status": "Failed", "startTime": "2026-09-22T07:28:38.432300499Z", "endTime": "2026-09-22T07:52:38.299308463Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svtah7m0vc8lt85cerhp2ucmjjj31gs still exists (deletion dispatched at 2026-09-22T07:28:40Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [svc] FrontendPathLatency fired 05:59:56 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T07:14:06Z Ended: 2026-09-22T07:47:06Z Severity: Sev3 Labels: alertname="FrontendPathLatency", cluster="ci01-j8719872-svc", component="frontend", method="get", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning" Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j8719872-svc). alert [svc] FrontendPathMedianLatency fired 05:59:56 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T07:13:07Z Ended: 2026-09-22T07:23:10Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j8719872-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j8719872-svc).
05:43:05	2h 20m 50s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head ad72b9c · merge ad72b9c	westus3	e2e	failed	#6945 (merged)	1	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 05:43:05 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000a3e1b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-t2xcxb7c2kws" to finish deleting: GET https://rp.j2229760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2409cd7b-f368-4677-b635-2e440f6713ab -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2409cd7b-f368-4677-b635-2e440f6713ab", "name": "2409cd7b-f368-4677-b635-2e440f6713ab", "status": "Failed", "startTime": "2026-09-22T07:22:36.959580513Z", "endTime": "2026-09-22T07:46:40.536487744Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svt7brd0t0cp9kf9q5at36rbcs9v3eb still exists (deletion dispatched at 2026-09-22T07:22:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000a3e1b0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-t2xcxb7c2kws" to finish deleting: GET https://rp.j2229760.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2409cd7b-f368-4677-b635-2e440f6713ab -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2409cd7b-f368-4677-b635-2e440f6713ab", "name": "2409cd7b-f368-4677-b635-2e440f6713ab", "status": "Failed", "startTime": "2026-09-22T07:22:36.959580513Z", "endTime": "2026-09-22T07:46:40.536487744Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svt7brd0t0cp9kf9q5at36rbcs9v3eb still exists (deletion dispatched at 2026-09-22T07:22:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
04:57:18	2h 9m 35s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head d1fadc5 · merge d1fadc5	westus3	n/a	passed	#7063 (merged)	0	n/a
04:31:30	38m 32s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 571559a · merge 571559a	centralus	provision	failed	★#6516 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code ConcurrentFederatedIdentityCredentialsWritesForSingleManagedIdentity; detail message Too many Federated Identity Credentials are written concurrently for the managed identity '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/microsoft.managedidentity/userassignedidentities/fleet'. Concurrent Federated Identity Credentials writes under the same managed identity are not supported. 04:31:30 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/cluster · suite templatize-pipeline Show raw failure time=2026-09-22T04:51:02.686Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster description="Step cluster\n Kind: ARM\n Template: templates/svc-cluster.bicep\n Parameters: configurations/svc-cluster.tmpl.bicepparam" time=2026-09-22T04:51:04.267Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster time=2026-09-22T04:51:06.089Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster deployment=96622666cca941a93b9a8651dce6827ae1f27fb7972b41f9caa78871db9bc985 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9675136-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F96622666cca941a93b9a8651dce6827ae1f27fb7972b41f9caa78871db9bc985 time=2026-09-22T05:04:09.339Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=cluster err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9675136-svc/providers/Microsoft.Resources/deployments/96622666cca941a93b9a8651dce6827ae1f27fb7972b41f9caa78871db9bc985/operationStatuses/08584115542204326289\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Conflict\",\n \"message\": \"{\\r\\n \\\"status\\\": \\\"Failed\\\",\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9675136-svc/providers/Microsoft.Resources/deployments/cluster-olgmgzuso436u\\\",\\r\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n \\\"details\\\": [\\r\\n {\\r\\n \\\"code\\\": \\\"ConcurrentFederatedIdentityCredentialsWritesForSingleManagedIdentity\\\",\\r\\n \\\"message\\\": \\\"Too many Federated Identity Credentials are written concurrently for the managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9675136-svc/providers/microsoft.managedidentity/userassignedidentities/fleet'. Concurrent Federated Identity Credentials writes under the same managed identity are not supported.\\\"\\r\\n }\\r\\n ]\\r\\n }\\r\\n ]\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n"
03:59:31	2h 23m 27s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head b80303c	canadacentral	alert, e2e	failed	#7060 (closed)	3	e2e (2) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 03:59:31 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00168a0a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-8kr9g49k2v48" to finish deleting: GET https://rp.j3806720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8fd82f3-9974-4eb6-997d-8bfaca303447 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8fd82f3-9974-4eb6-997d-8bfaca303447", "name": "c8fd82f3-9974-4eb6-997d-8bfaca303447", "status": "Failed", "startTime": "2026-09-22T05:41:49.238086476Z", "endTime": "2026-09-22T06:05:52.932440268Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svrq1s2u91i6dulrlshbntsg0um43o3 still exists (deletion dispatched at 2026-09-22T05:41:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00168a0a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-8kr9g49k2v48" to finish deleting: GET https://rp.j3806720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8fd82f3-9974-4eb6-997d-8bfaca303447 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c8fd82f3-9974-4eb6-997d-8bfaca303447", "name": "c8fd82f3-9974-4eb6-997d-8bfaca303447", "status": "Failed", "startTime": "2026-09-22T05:41:49.238086476Z", "endTime": "2026-09-22T06:05:52.932440268Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svrq1s2u91i6dulrlshbntsg0um43o3 still exists (deletion dispatched at 2026-09-22T05:41:50Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 03:59:31 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster Unexpected error: <fmt.wrapError \| 0xc001342060>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-8sb2nhgvgtbz" to finish creating: GET https://rp.j3806720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0fbffb8b-7479-448a-86b5-beac660cad46 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0fbffb8b-7479-448a-86b5-beac660cad46", "name": "0fbffb8b-7479-448a-86b5-beac660cad46", "status": "Failed", "startTime": "2026-09-22T05:08:13.528034946Z", "endTime": "2026-09-22T05:27:17.856254463Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster Unexpected error: <fmt.wrapError \| 0xc001342060>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-8sb2nhgvgtbz" to finish creating: GET https://rp.j3806720.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0fbffb8b-7479-448a-86b5-beac660cad46 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0fbffb8b-7479-448a-86b5-beac660cad46", "name": "0fbffb8b-7479-448a-86b5-beac660cad46", "status": "Failed", "startTime": "2026-09-22T05:08:13.528034946Z", "endTime": "2026-09-22T05:27:17.856254463Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] BackendControllerQueueDepthHigh fired 03:59:31 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T05:14:22Z Ended: 2026-09-22T05:26:22Z Severity: Sev3 Labels: alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3806720-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning" Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
03:19:01	2h 19m 17s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head d8bb7ef · merge d8bb7ef	centralus	alert, e2e	failed	#6891 (merged)	2	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 03:19:01 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0016cc0d8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-hx6p7dn6kkf4" to finish deleting: GET https://rp.j7837312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8738611c-b07f-435d-88e1-22988aa39656 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8738611c-b07f-435d-88e1-22988aa39656", "name": "8738611c-b07f-435d-88e1-22988aa39656", "status": "Failed", "startTime": "2026-09-22T04:58:16.745509815Z", "endTime": "2026-09-22T05:22:18.875705001Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svr2l1g09bfh9s4k79s1trmkkm0nli7 still exists (deletion dispatched at 2026-09-22T04:58:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0016cc0d8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-hx6p7dn6kkf4" to finish deleting: GET https://rp.j7837312.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8738611c-b07f-435d-88e1-22988aa39656 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/8738611c-b07f-435d-88e1-22988aa39656", "name": "8738611c-b07f-435d-88e1-22988aa39656", "status": "Failed", "startTime": "2026-09-22T04:58:16.745509815Z", "endTime": "2026-09-22T05:22:18.875705001Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svr2l1g09bfh9s4k79s1trmkkm0nli7 still exists (deletion dispatched at 2026-09-22T04:58:18Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] ServiceMemoryDrift fired 03:19:01 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T05:26:58Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j7837312-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-g5l6j", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-g5l6j (namespace arobit) on cluster ci01-j7837312-svc has averaged 126.4% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
03:00:48	2h 4m 19s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head ed561d7 · merge ed561d7	westus3	n/a	passed	#7042 (merged)	0	n/a
03:00:43	2h 18m 20s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 1c995b8	westus3	e2e	failed	#7053 (open)	1	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 03:00:43 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a24048>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-6stmg9wlrf72" to finish deleting: GET https://rp.j8290944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/11876048-7a18-472b-ae65-ac19a08a4a96 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/11876048-7a18-472b-ae65-ac19a08a4a96", "name": "11876048-7a18-472b-ae65-ac19a08a4a96", "status": "Failed", "startTime": "2026-09-22T04:37:24.350700065Z", "endTime": "2026-09-22T05:01:26.139604116Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svqp7o4idcshklg9rpdsdv2sclsb1mm still exists (deletion dispatched at 2026-09-22T04:37:25Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a24048>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-6stmg9wlrf72" to finish deleting: GET https://rp.j8290944.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/11876048-7a18-472b-ae65-ac19a08a4a96 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/11876048-7a18-472b-ae65-ac19a08a4a96", "name": "11876048-7a18-472b-ae65-ac19a08a4a96", "status": "Failed", "startTime": "2026-09-22T04:37:24.350700065Z", "endTime": "2026-09-22T05:01:26.139604116Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svqp7o4idcshklg9rpdsdv2sclsb1mm still exists (deletion dispatched at 2026-09-22T04:37:25Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
03:00:33	2h 17m 15s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head e8df860 · merge e8df860	canadacentral	alert, e2e	failed	#5846 (merged)	3	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 03:00:33 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00178c138>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-dllnjkc42xws" to finish deleting: GET https://rp.j1728768.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a2f2f18c-0a11-4b43-86e6-22cd37fa6dd4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a2f2f18c-0a11-4b43-86e6-22cd37fa6dd4", "name": "a2f2f18c-0a11-4b43-86e6-22cd37fa6dd4", "status": "Failed", "startTime": "2026-09-22T04:36:26.910866439Z", "endTime": "2026-09-22T05:00:27.577656556Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svqoe5e9dp3g5ri39rlt9i6u9dmclsp still exists (deletion dispatched at 2026-09-22T04:36:28Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc00178c138>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-dllnjkc42xws" to finish deleting: GET https://rp.j1728768.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a2f2f18c-0a11-4b43-86e6-22cd37fa6dd4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a2f2f18c-0a11-4b43-86e6-22cd37fa6dd4", "name": "a2f2f18c-0a11-4b43-86e6-22cd37fa6dd4", "status": "Failed", "startTime": "2026-09-22T04:36:26.910866439Z", "endTime": "2026-09-22T05:00:27.577656556Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svqoe5e9dp3g5ri39rlt9i6u9dmclsp still exists (deletion dispatched at 2026-09-22T04:36:28Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (2) alert [svc] FrontendPathLatency fired 03:00:33 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T04:17:50Z Ended: 2026-09-22T04:50:51Z Severity: Sev3 Labels: alertname="FrontendPathLatency", cluster="ci01-j1728768-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning" Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j1728768-svc). alert [svc] ServiceMemoryDrift fired 03:00:33 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T04:51:26Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j1728768-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-wdz85", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-wdz85 (namespace arobit) on cluster ci01-j1728768-svc has averaged 125.8% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
03:00:25	2h 24m 49s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head ad72b9c · merge ad72b9c	canadacentral	e2e	failed	#6945 (merged)	4	e2e (4) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 03:00:25 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001624078>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-65cd4v" in resourcegroup="rg-cp-ystream-upgrade-4-22-p6445jwrw8px" to finish deleting: GET https://rp.j9524608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a8da0d4c-832a-4ebe-83fe-0ef1ea20c2ed -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a8da0d4c-832a-4ebe-83fe-0ef1ea20c2ed", "name": "a8da0d4c-832a-4ebe-83fe-0ef1ea20c2ed", "status": "Failed", "startTime": "2026-09-22T04:21:02.250571798Z", "endTime": "2026-09-22T04:45:04.991809002Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svqpgqbhn074kn1grepral244a0uo9a still exists (deletion dispatched at 2026-09-22T04:21:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001624078>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-22-65cd4v" in resourcegroup="rg-cp-ystream-upgrade-4-22-p6445jwrw8px" to finish deleting: GET https://rp.j9524608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a8da0d4c-832a-4ebe-83fe-0ef1ea20c2ed -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a8da0d4c-832a-4ebe-83fe-0ef1ea20c2ed", "name": "a8da0d4c-832a-4ebe-83fe-0ef1ea20c2ed", "status": "Failed", "startTime": "2026-09-22T04:21:02.250571798Z", "endTime": "2026-09-22T04:45:04.991809002Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svqpgqbhn074kn1grepral244a0uo9a still exists (deletion dispatched at 2026-09-22T04:21:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 03:00:25 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0015ee120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-21-dvbs4m" in resourcegroup="rg-cp-ystream-upgrade-4-21-8dfdxdjq6sl7" to finish deleting: GET https://rp.j9524608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dfeb1299-93f0-4935-b7a6-3aa4e0368f83 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dfeb1299-93f0-4935-b7a6-3aa4e0368f83", "name": "dfeb1299-93f0-4935-b7a6-3aa4e0368f83", "status": "Failed", "startTime": "2026-09-22T04:21:06.047683642Z", "endTime": "2026-09-22T04:45:14.955489839Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svqpgvvjijkhsd79om9jjphnksski1h still exists (deletion dispatched at 2026-09-22T04:21:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0015ee120>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cp-ystream-upgrade-4-21-dvbs4m" in resourcegroup="rg-cp-ystream-upgrade-4-21-8dfdxdjq6sl7" to finish deleting: GET https://rp.j9524608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dfeb1299-93f0-4935-b7a6-3aa4e0368f83 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/dfeb1299-93f0-4935-b7a6-3aa4e0368f83", "name": "dfeb1299-93f0-4935-b7a6-3aa4e0368f83", "status": "Failed", "startTime": "2026-09-22T04:21:06.047683642Z", "endTime": "2026-09-22T04:45:14.955489839Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svqpgvvjijkhsd79om9jjphnksski1h still exists (deletion dispatched at 2026-09-22T04:21:08Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 03:00:25 UTC · failed at e2e · test Customer should be able to update nodepool replicas and autoscaling · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001074030>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-4pj899qpn6jc" to finish deleting: GET https://rp.j9524608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7b12757b-9d14-428d-b673-87c01137c0b4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7b12757b-9d14-428d-b673-87c01137c0b4", "name": "7b12757b-9d14-428d-b673-87c01137c0b4", "status": "Failed", "startTime": "2026-09-22T04:32:27.654860692Z", "endTime": "2026-09-22T04:56:34.502862674Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svqpiu74flsdu8i1gba51b7sq6qn8pc still exists (deletion dispatched at 2026-09-22T04:32:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001074030>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-4pj899qpn6jc" to finish deleting: GET https://rp.j9524608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7b12757b-9d14-428d-b673-87c01137c0b4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/7b12757b-9d14-428d-b673-87c01137c0b4", "name": "7b12757b-9d14-428d-b673-87c01137c0b4", "status": "Failed", "startTime": "2026-09-22T04:32:27.654860692Z", "endTime": "2026-09-22T04:56:34.502862674Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svqpiu74flsdu8i1gba51b7sq6qn8pc still exists (deletion dispatched at 2026-09-22T04:32:29Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters, 2 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 2 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 03:00:25 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a62078>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-g5q4pnrvzzcw" to finish deleting: GET https://rp.j9524608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cebb22bc-25cc-4fe0-9947-7179f9b8772b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cebb22bc-25cc-4fe0-9947-7179f9b8772b", "name": "cebb22bc-25cc-4fe0-9947-7179f9b8772b", "status": "Failed", "startTime": "2026-09-22T04:43:42.181198275Z", "endTime": "2026-09-22T05:07:44.535589725Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svqpft6nt92r6veu7aabqm40942hic4 still exists (deletion dispatched at 2026-09-22T04:43:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a62078>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-g5q4pnrvzzcw" to finish deleting: GET https://rp.j9524608.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cebb22bc-25cc-4fe0-9947-7179f9b8772b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/cebb22bc-25cc-4fe0-9947-7179f9b8772b", "name": "cebb22bc-25cc-4fe0-9947-7179f9b8772b", "status": "Failed", "startTime": "2026-09-22T04:43:42.181198275Z", "endTime": "2026-09-22T05:07:44.535589725Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svqpft6nt92r6veu7aabqm40942hic4 still exists (deletion dispatched at 2026-09-22T04:43:44Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
02:56:33	2h 13m 16s	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 016fdec · merge 02f5585	centralus	e2e	failed	#5724 (merged)	1	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 02:56:33 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001262150>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-mm8687thvtjz" to finish deleting: GET https://rp.j0337024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b787e318-ad3e-4acc-bfcf-e4f4c39ce55a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b787e318-ad3e-4acc-bfcf-e4f4c39ce55a", "name": "b787e318-ad3e-4acc-bfcf-e4f4c39ce55a", "status": "Failed", "startTime": "2026-09-22T04:30:32.899691516Z", "endTime": "2026-09-22T04:54:37.595087082Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svqmjoqgfjcv3n68g00396ud2hcevk5 still exists (deletion dispatched at 2026-09-22T04:30:34Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001262150>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-mm8687thvtjz" to finish deleting: GET https://rp.j0337024.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b787e318-ad3e-4acc-bfcf-e4f4c39ce55a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/b787e318-ad3e-4acc-bfcf-e4f4c39ce55a", "name": "b787e318-ad3e-4acc-bfcf-e4f4c39ce55a", "status": "Failed", "startTime": "2026-09-22T04:30:32.899691516Z", "endTime": "2026-09-22T04:54:37.595087082Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svqmjoqgfjcv3n68g00396ud2hcevk5 still exists (deletion dispatched at 2026-09-22T04:30:34Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
02:55:57	2h 26m 3s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 95f6c7a · merge 95f6c7a	westus3	alert, e2e	failed	#6948 (merged)	2	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 02:55:57 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0012dc0a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-sw29fbqg4jf9" to finish deleting: GET https://rp.j8736640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a5c5a2a-f5fb-43d1-be90-346fb5dab267 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a5c5a2a-f5fb-43d1-be90-346fb5dab267", "name": "5a5c5a2a-f5fb-43d1-be90-346fb5dab267", "status": "Failed", "startTime": "2026-09-22T04:41:00.185282796Z", "endTime": "2026-09-22T05:05:08.702721718Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svqqrb0hjk2rjgf2447lvn1fit1l09o still exists (deletion dispatched at 2026-09-22T04:41:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0012dc0a8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-sw29fbqg4jf9" to finish deleting: GET https://rp.j8736640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a5c5a2a-f5fb-43d1-be90-346fb5dab267 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/5a5c5a2a-f5fb-43d1-be90-346fb5dab267", "name": "5a5c5a2a-f5fb-43d1-be90-346fb5dab267", "status": "Failed", "startTime": "2026-09-22T04:41:00.185282796Z", "endTime": "2026-09-22T05:05:08.702721718Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svqqrb0hjk2rjgf2447lvn1fit1l09o still exists (deletion dispatched at 2026-09-22T04:41:01Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] FrontendPathMedianLatency fired 02:55:57 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T03:59:08Z Ended: 2026-09-22T04:33:08Z Severity: Sev3 Labels: alertname="FrontendPathMedianLatency", cluster="ci01-j8736640-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j8736640-svc).
02:42:18	42m 56s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head c608bc0 · merge c608bc0	centralus	provision	failed	★#7062 (merged)	1	provision (1) ERROR CODE: Conflict; detail code ObjectIsDeletedButRecoverable 02:42:18 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/admin-api-cert · suite templatize-pipeline Show raw failure time=2026-09-22T03:15:16.448Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert description="Step admin-api-cert\n Kind: CreateCertificate\n" time=2026-09-22T03:15:16.448Z level=INFO msg="Creating certificate" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert vaultBaseUrl=https://aro-hcp-dev-svc-kv.vault.azure.net/ certificateName=admin-api-cert-ci01-j2691712 issuer=Self commonName=admin.j2691712.hcpsvc.osadev.cloud san=admin.j2691712.hcpsvc.osadev.cloud time=2026-09-22T03:15:17.222Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert err="error running Create Certificate Step: failed to create certificate \"admin-api-cert-ci01-j2691712\" in vault \"https://aro-hcp-dev-svc-kv.vault.azure.net/\": POST https://aro-hcp-dev-svc-kv.vault.azure.net/certificates/admin-api-cert-ci01-j2691712/create\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Certificate admin-api-cert-ci01-j2691712 is currently in a deleted but recoverable state, and its name cannot be reused; in this state, the certificate can only be recovered or purged.\",\n \"innererror\": {\n \"code\": \"ObjectIsDeletedButRecoverable\"\n }\n }\n}\n--------------------------------------------------------------------------------\n"
01:38:36	2h 7m 51s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head b80303c	canadacentral	alert, e2e	failed	#7060 (closed)	4	e2e (3) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 01:38:36 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a441e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-gkxv2rdhffbn" to finish deleting: GET https://rp.j2947968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/539c9b60-22fe-4178-8a04-19adb0069261 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/539c9b60-22fe-4178-8a04-19adb0069261", "name": "539c9b60-22fe-4178-8a04-19adb0069261", "status": "Failed", "startTime": "2026-09-22T03:04:45.070729658Z", "endTime": "2026-09-22T03:28:49.639377463Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svpfbrcndrja8lh6o4gbf38h9p1ph6m still exists (deletion dispatched at 2026-09-22T03:04:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001a441e0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-gkxv2rdhffbn" to finish deleting: GET https://rp.j2947968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/539c9b60-22fe-4178-8a04-19adb0069261 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/539c9b60-22fe-4178-8a04-19adb0069261", "name": "539c9b60-22fe-4178-8a04-19adb0069261", "status": "Failed", "startTime": "2026-09-22T03:04:45.070729658Z", "endTime": "2026-09-22T03:28:49.639377463Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svpfbrcndrja8lh6o4gbf38h9p1ph6m still exists (deletion dispatched at 2026-09-22T03:04:47Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: nodePools, serviceProviderClusters, serviceProviderNodePools; provider Microsoft.RedHatOpenShift 01:38:36 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0015a65b8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-skip-minor-sjts2p" in resourcegroup="rg-np-skip-minor-sjts2p-xgpmp2tpf5bv" to finish deleting: GET https://rp.j2947968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d26277a6-0af9-4da0-80f5-448652226fff -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d26277a6-0af9-4da0-80f5-448652226fff", "name": "d26277a6-0af9-4da0-80f5-448652226fff", "status": "Failed", "startTime": "2026-09-22T03:01:26.657883091Z", "endTime": "2026-09-22T03:25:29.637165858Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svpfbucgbjrpto0mrulh6vn6q4os1c1 still exists (deletion dispatched at 2026-09-22T03:01:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0015a65b8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-skip-minor-sjts2p" in resourcegroup="rg-np-skip-minor-sjts2p-xgpmp2tpf5bv" to finish deleting: GET https://rp.j2947968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d26277a6-0af9-4da0-80f5-448652226fff -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d26277a6-0af9-4da0-80f5-448652226fff", "name": "d26277a6-0af9-4da0-80f5-448652226fff", "status": "Failed", "startTime": "2026-09-22T03:01:26.657883091Z", "endTime": "2026-09-22T03:25:29.637165858Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svpfbucgbjrpto0mrulh6vn6q4os1c1 still exists (deletion dispatched at 2026-09-22T03:01:27Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 Microsoft.RedHatOpenShift/hcpOpenShiftClusters/nodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/nodepools/serviceProviderNodePools, 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 01:38:36 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000b302a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-rmtxh2ts9kfb" to finish deleting: GET https://rp.j2947968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/586ad5d7-78c5-445c-8754-3de3e7992633 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/586ad5d7-78c5-445c-8754-3de3e7992633", "name": "586ad5d7-78c5-445c-8754-3de3e7992633", "status": "Failed", "startTime": "2026-09-22T02:50:05.627831093Z", "endTime": "2026-09-22T03:14:06.922141251Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svpfn4442k3esfkats1kkvuno9gqgd0 still exists (deletion dispatched at 2026-09-22T02:50:07Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc000b302a0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-rmtxh2ts9kfb" to finish deleting: GET https://rp.j2947968.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/586ad5d7-78c5-445c-8754-3de3e7992633 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/586ad5d7-78c5-445c-8754-3de3e7992633", "name": "586ad5d7-78c5-445c-8754-3de3e7992633", "status": "Failed", "startTime": "2026-09-22T02:50:05.627831093Z", "endTime": "2026-09-22T03:14:06.922141251Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svpfn4442k3esfkats1kkvuno9gqgd0 still exists (deletion dispatched at 2026-09-22T02:50:07Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] ServiceMemoryDrift fired 01:38:36 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T03:24:14Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j2947968-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-49lgs", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-49lgs (namespace arobit) on cluster ci01-j2947968-svc has averaged 126.8% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
01:10:26	2h 1m 40s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head 52b8f22 · merge 52b8f22	canadacentral	e2e	failed	#7061 (merged)	1	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 01:10:26 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0013ac0c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-bx6lqvftk4w7" to finish deleting: GET https://rp.j8244736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/151984c8-dc03-45fe-a98d-c1ec95b80b1b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/151984c8-dc03-45fe-a98d-c1ec95b80b1b", "name": "151984c8-dc03-45fe-a98d-c1ec95b80b1b", "status": "Failed", "startTime": "2026-09-22T02:31:50.927530568Z", "endTime": "2026-09-22T02:55:59.602648065Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svp6bj50tirtgu0edlj3fjaggm8e3pr still exists (deletion dispatched at 2026-09-22T02:31:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0013ac0c0>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-bx6lqvftk4w7" to finish deleting: GET https://rp.j8244736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/151984c8-dc03-45fe-a98d-c1ec95b80b1b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/151984c8-dc03-45fe-a98d-c1ec95b80b1b", "name": "151984c8-dc03-45fe-a98d-c1ec95b80b1b", "status": "Failed", "startTime": "2026-09-22T02:31:50.927530568Z", "endTime": "2026-09-22T02:55:59.602648065Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svp6bj50tirtgu0edlj3fjaggm8e3pr still exists (deletion dispatched at 2026-09-22T02:31:53Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred
00:21:24	2h 13m 12s	pull-ci-Azure-ARO-HCP-main-e2e-parallel merged PR head 7d175e1 · merge b22cddf	westus3	alert, e2e	failed	#7059 (merged)	2	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 00:21:24 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0011d4108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-rlrmtm9g69c6" to finish deleting: GET https://rp.j1868672.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b5f8d503-82d0-44c5-b405-81069f91db8c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b5f8d503-82d0-44c5-b405-81069f91db8c", "name": "b5f8d503-82d0-44c5-b405-81069f91db8c", "status": "Failed", "startTime": "2026-09-22T01:52:36.957024286Z", "endTime": "2026-09-22T02:16:41.168615942Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svocvbp6291hdqpj2pim2m0gvn0ml5d still exists (deletion dispatched at 2026-09-22T01:52:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0011d4108>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-rlrmtm9g69c6" to finish deleting: GET https://rp.j1868672.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b5f8d503-82d0-44c5-b405-81069f91db8c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b5f8d503-82d0-44c5-b405-81069f91db8c", "name": "b5f8d503-82d0-44c5-b405-81069f91db8c", "status": "Failed", "startTime": "2026-09-22T01:52:36.957024286Z", "endTime": "2026-09-22T02:16:41.168615942Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svocvbp6291hdqpj2pim2m0gvn0ml5d still exists (deletion dispatched at 2026-09-22T01:52:41Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] ServiceMemoryDrift fired 00:21:24 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T02:22:33Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j1868672-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-x2z5m", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-x2z5m (namespace arobit) on cluster ci01-j1868672-svc has averaged 126.3% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
00:02:48	36m 15s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head ed561d7 · merge ed561d7	centralus	provision	failed	★#7042 (merged)	1	provision (1) ERROR CODE: Conflict; detail code ObjectIsDeletedButRecoverable 00:02:48 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/service/admin-api-cert · suite templatize-pipeline Show raw failure time=2026-09-22T00:26:51.703Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert description="Step admin-api-cert\n Kind: CreateCertificate\n" time=2026-09-22T00:26:51.703Z level=INFO msg="Creating certificate" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert vaultBaseUrl=https://aro-hcp-dev-svc-kv.vault.azure.net/ certificateName=admin-api-cert-ci01-j1796608 issuer=Self commonName=admin.j1796608.hcpsvc.osadev.cloud san=admin.j1796608.hcpsvc.osadev.cloud time=2026-09-22T00:26:52.483Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=admin-api-cert err="error running Create Certificate Step: failed to create certificate \"admin-api-cert-ci01-j1796608\" in vault \"https://aro-hcp-dev-svc-kv.vault.azure.net/\": POST https://aro-hcp-dev-svc-kv.vault.azure.net/certificates/admin-api-cert-ci01-j1796608/create\n--------------------------------------------------------------------------------\nRESPONSE 409: 409 Conflict\nERROR CODE: Conflict\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"Conflict\",\n \"message\": \"Certificate admin-api-cert-ci01-j1796608 is currently in a deleted but recoverable state, and its name cannot be reused; in this state, the certificate can only be recovered or purged.\",\n \"innererror\": {\n \"code\": \"ObjectIsDeletedButRecoverable\"\n }\n }\n}\n--------------------------------------------------------------------------------\n"
00:02:45	2h 11m 13s	pull-ci-Azure-ARO-HCP-main-e2e-parallel head 1c995b8	centralus	alert, e2e	failed	#7053 (open)	2	e2e (1) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 00:02:45 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0012c20d8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-dg-minor-frdg8x" in resourcegroup="rg-np-dg-minor-frdg8x-bmvs69lrn9nj" to finish deleting: GET https://rp.j3356288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e0f3f148-543d-4e5c-9517-ece99016d1a5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e0f3f148-543d-4e5c-9517-ece99016d1a5", "name": "e0f3f148-543d-4e5c-9517-ece99016d1a5", "status": "Failed", "startTime": "2026-09-22T01:33:09.939984258Z", "endTime": "2026-09-22T01:57:11.486464494Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svo54ortklo3gj9q24qi42opdhtistm still exists (deletion dispatched at 2026-09-22T01:33:11Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc0012c20d8>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-dg-minor-frdg8x" in resourcegroup="rg-np-dg-minor-frdg8x-bmvs69lrn9nj" to finish deleting: GET https://rp.j3356288.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e0f3f148-543d-4e5c-9517-ece99016d1a5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e0f3f148-543d-4e5c-9517-ece99016d1a5", "name": "e0f3f148-543d-4e5c-9517-ece99016d1a5", "status": "Failed", "startTime": "2026-09-22T01:33:09.939984258Z", "endTime": "2026-09-22T01:57:11.486464494Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svo54ortklo3gj9q24qi42opdhtistm still exists (deletion dispatched at 2026-09-22T01:33:11Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] FrontendPathLatency fired 00:02:45 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathLatency does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Resolved Started: 2026-09-22T01:34:08Z Ended: 2026-09-22T01:43:06Z Severity: Sev3 Labels: alertname="FrontendPathLatency", cluster="ci01-j3356288-svc", component="frontend", method="delete", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning" Description: More than 5% of frontend requests for delete /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j3356288-svc).
00:02:29	2h 15m 6s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head ad72b9c · merge ad72b9c	canadacentral	alert, e2e	failed	#6945 (merged)	3	e2e (2) ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 00:02:29 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001404e70>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-xjshj2" in resourcegroup="rg-np-version-upgrade-xjshj2-h4xgk6nl5j9d" to finish deleting: GET https://rp.j8483200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1a98cfab-0566-40a0-9178-58b4073d509a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1a98cfab-0566-40a0-9178-58b4073d509a", "name": "1a98cfab-0566-40a0-9178-58b4073d509a", "status": "Failed", "startTime": "2026-09-22T01:33:50.788314397Z", "endTime": "2026-09-22T01:57:55.348577083Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svo4230k0qs66paonqn2e2dbtlcusvd still exists (deletion dispatched at 2026-09-22T01:33:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001404e70>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="np-version-upgrade-cluster-xjshj2" in resourcegroup="rg-np-version-upgrade-xjshj2-h4xgk6nl5j9d" to finish deleting: GET https://rp.j8483200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1a98cfab-0566-40a0-9178-58b4073d509a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1a98cfab-0566-40a0-9178-58b4073d509a", "name": "1a98cfab-0566-40a0-9178-58b4073d509a", "status": "Failed", "startTime": "2026-09-22T01:33:50.788314397Z", "endTime": "2026-09-22T01:57:55.348577083Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svo4230k0qs66paonqn2e2dbtlcusvd still exists (deletion dispatched at 2026-09-22T01:33:52Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift 00:02:29 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001880030>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-w8lc4ztvgcdz" to finish deleting: GET https://rp.j8483200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ab28a42-8341-418d-96d7-128339122a83 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ab28a42-8341-418d-96d7-128339122a83", "name": "9ab28a42-8341-418d-96d7-128339122a83", "status": "Failed", "startTime": "2026-09-22T01:36:01.895564169Z", "endTime": "2026-09-22T02:00:05.356533198Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svo404qsimsek2clk2adqccvekdokcj still exists (deletion dispatched at 2026-09-22T01:36:03Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error: <errors.joinError \| 0xc001880030>: failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-w8lc4ztvgcdz" to finish deleting: GET https://rp.j8483200.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ab28a42-8341-418d-96d7-128339122a83 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9ab28a42-8341-418d-96d7-128339122a83", "name": "9ab28a42-8341-418d-96d7-128339122a83", "status": "Failed", "startTime": "2026-09-22T01:36:01.895564169Z", "endTime": "2026-09-22T02:00:05.356533198Z", "error": { "code": "InternalServerError", "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2svo404qsimsek2clk2adqccvekdokcj still exists (deletion dispatched at 2026-09-22T01:36:03Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters" } } -------------------------------------------------------------------------------- ... occurred alert (1) alert [svc] ServiceMemoryDrift fired 00:02:29 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests Show raw failure alert fired 1 time(s) Firing 1: State: Fired (not resolved) Started: 2026-09-22T01:58:50Z Severity: Sev3 Labels: alertname="ServiceMemoryDrift", cluster="ci01-j8483200-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-pgmn8", severity="warning", team="hcp-sl" Description: Container fluentbit in pod arobit-forwarder-pgmn8 (namespace arobit) on cluster ci01-j8483200-svc has averaged 126.2% of its memory request over the last 30 minutes. This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
00:02:27	40m 41s	pull-ci-Azure-ARO-HCP-main-e2e-parallel post-goodmerged PR head e8df860 · merge e8df860	centralus	provision	failed	★#5846 (merged)	1	provision (1) ERROR CODE: DeploymentFailed; detail code ReconcileVMSSAgentPoolFailed; provider Microsoft.ContainerService 00:02:27 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/cluster-create (stamp=1) · suite templatize-pipeline Show raw failure time=2026-09-22T00:28:42.028Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 description="Step cluster-create\n Kind: Shell\n Command: ./aks-cluster-create\n" time=2026-09-22T00:31:19.046Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=1 err="stamp 1: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-22T00:28:42.035099926Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5677184-mgmt-1\",\"clusterName\":\"ci01-j5677184-mgmt-1\",\"region\":\"centralus\"}\n{\"time\":\"2026-09-22T00:28:44.199184892Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5677184-mgmt-1\",\"clusterName\":\"ci01-j5677184-mgmt-1\",\"region\":\"centralus\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-22T00:28:44.294775263Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5677184-mgmt-1\",\"clusterName\":\"ci01-j5677184-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-22T00:31:19.041004649Z\",\"level\":\"INFO\",\"msg\":\"deployment operation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5677184-mgmt-1\",\"clusterName\":\"ci01-j5677184-mgmt-1\",\"region\":\"centralus\",\"deployment\":\"aks-cluster-create\",\"operationID\":\"B5981CAA76057630\",\"resource\":{\"id\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5677184-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5677184-mgmt-1\",\"resourceName\":\"ci01-j5677184-mgmt-1\",\"resourceType\":\"Microsoft.ContainerService/managedClusters\"},\"provisioningOperation\":\"Create\",\"provisioningState\":\"Failed\",\"timestamp\":\"2026-09-22T00:28:49.2033491Z\",\"duration\":\"PT2M28.4887595S\",\"statusCode\":\"Conflict\",\"statusMessage\":{\"error\":{\"code\":\"ReconcileVMSSAgentPoolFailed\",\"message\":\"Internal server error\"},\"status\":\"Failed\"}}\n{\"time\":\"2026-09-22T00:31:19.041107186Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5677184-mgmt-1\",\"clusterName\":\"ci01-j5677184-mgmt-1\",\"region\":\"centralus\",\"err\":\"waiting for deployment aks-cluster-create: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5677184-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584115699609930281\\n--------------------------------------------------------------------------------\\nRESPONSE 200: 200 OK\\nERROR CODE: DeploymentFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"status\\\": \\\"Failed\\\",\\n \\\"error\\\": {\\n \\\"code\\\": \\\"DeploymentFailed\\\",\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5677184-mgmt-1/providers/Microsoft.Resources/deployments/aks-cluster-create\\\",\\n \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\n \\\"details\\\": [\\n {\\n \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\n \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j5677184-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j5677184-mgmt-1\\\",\\n \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\n \\\"details\\\": [\\n {\\n \\\"code\\\": \\\"ReconcileVMSSAgentPoolFailed\\\",\\n \\\"message\\\": \\\"Internal server error\\\"\\n }\\n ]\\n }\\n ]\\n }\\n}\\n--------------------------------------------------------------------------------\\n\",\"elapsed\":\"2m37.00601594s\"}\n exit status 1"

Environment: INT

Time (UTC)	Runtime	Job	Region	Failed at	Result	PR	Failed tests	Details
19:50:05	3h 4m 46s	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel post-goodmerged PR No additional run metadata captured	westus3	e2e	failed	n/a	3	e2e (3) ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 19:50:05 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-h7c9rnh9n24x/cluster-candidate-4-21-jthhdc should provision Unexpected error: <fmt.wrapError \| 0xc00102a600>: failed to create HCP cluster cluster-candidate-4-21-jthhdc: failed waiting for cluster="cluster-candidate-4-21-jthhdc" in resourcegroup="rg-candidate-4-21-h7c9rnh9n24x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b55cb22c-9a55-40c5-b3c6-893ec03a9fc6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b55cb22c-9a55-40c5-b3c6-893ec03a9fc6", "name": "b55cb22c-9a55-40c5-b3c6-893ec03a9fc6", "status": "Failed", "startTime": "2026-09-22T21:07:20.636049246Z", "endTime": "2026-09-22T21:36:27.958431734Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 2m58s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-h7c9rnh9n24x/cluster-candidate-4-21-jthhdc should provision Unexpected error: <fmt.wrapError \| 0xc00102a600>: failed to create HCP cluster cluster-candidate-4-21-jthhdc: failed waiting for cluster="cluster-candidate-4-21-jthhdc" in resourcegroup="rg-candidate-4-21-h7c9rnh9n24x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b55cb22c-9a55-40c5-b3c6-893ec03a9fc6 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b55cb22c-9a55-40c5-b3c6-893ec03a9fc6", "name": "b55cb22c-9a55-40c5-b3c6-893ec03a9fc6", "status": "Failed", "startTime": "2026-09-22T21:07:20.636049246Z", "endTime": "2026-09-22T21:36:27.958431734Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 2m58s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred failed to locate cilium chart: Get "<url>": dial tcp <ip>: connect: network is unreachable 19:50:05 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:159]: failed to install Cilium chart via Helm Unexpected error: <fmt.wrapError \| 0xc0008581e0>: failed to locate cilium chart: Get "https://helm.cilium.io/cilium-1.19.2.tgz": dial tcp [2606:50c0:8000::153]:443: connect: network is unreachable ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:159]: failed to install Cilium chart via Helm Unexpected error: <fmt.wrapError \| 0xc0008581e0>: failed to locate cilium chart: Get "https://helm.cilium.io/cilium-1.19.2.tgz": dial tcp [2606:50c0:8000::153]:443: connect: network is unreachable ... occurred ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift 19:50:05 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite integration/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000914940>: failed to create HCP cluster ea-list7cctv9: failed waiting for cluster="ea-list7cctv9" in resourcegroup="ea-list-wc6fbxsd9rxx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3a3e500-1ba1-4cb7-8434-bc1aa96ee070 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3a3e500-1ba1-4cb7-8434-bc1aa96ee070", "name": "c3a3e500-1ba1-4cb7-8434-bc1aa96ee070", "status": "Failed", "startTime": "2026-09-22T21:22:56.593640254Z", "endTime": "2026-09-22T21:51:58.372098632Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 2m51s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000914940>: failed to create HCP cluster ea-list7cctv9: failed waiting for cluster="ea-list7cctv9" in resourcegroup="ea-list-wc6fbxsd9rxx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3a3e500-1ba1-4cb7-8434-bc1aa96ee070 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3a3e500-1ba1-4cb7-8434-bc1aa96ee070", "name": "c3a3e500-1ba1-4cb7-8434-bc1aa96ee070", "status": "Failed", "startTime": "2026-09-22T21:22:56.593640254Z", "endTime": "2026-09-22T21:51:58.372098632Z", "error": { "code": "InternalServerError", "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 2m51s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas" } } -------------------------------------------------------------------------------- ... occurred
16:24:26	2h 13m 13s	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel post-goodmerged PR No additional run metadata captured	westus3	n/a	passed	n/a	0	n/a
13:11:15	2h 28m 25s	branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel post-goodmerged PR No additional run metadata captured	westus3	n/a	passed	n/a	0	n/a

Time (UTC)

Runtime

Job

Region

Failed at

Result

Failed tests

Details

19:50:05

3h 4m 46s

branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel

post-goodmerged PR

No additional run metadata captured

westus3

e2e

failed

n/a

e2e (3)

ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [ignition-server-proxy]; provider Microsoft.RedHatOpenShift

19:50:05 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-h7c9rnh9n24x/cluster-candidate-4-21-jthhdc should provision
Unexpected error:
    <*fmt.wrapError | 0xc00102a600>: 
    failed to create HCP cluster cluster-candidate-4-21-jthhdc: failed waiting for cluster="cluster-candidate-4-21-jthhdc" in resourcegroup="rg-candidate-4-21-h7c9rnh9n24x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b55cb22c-9a55-40c5-b3c6-893ec03a9fc6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b55cb22c-9a55-40c5-b3c6-893ec03a9fc6",
      "name": "b55cb22c-9a55-40c5-b3c6-893ec03a9fc6",
      "status": "Failed",
      "startTime": "2026-09-22T21:07:20.636049246Z",
      "endTime": "2026-09-22T21:36:27.958431734Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 2m58s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-h7c9rnh9n24x/cluster-candidate-4-21-jthhdc should provision
Unexpected error:
    <*fmt.wrapError | 0xc00102a600>: 
    failed to create HCP cluster cluster-candidate-4-21-jthhdc: failed waiting for cluster="cluster-candidate-4-21-jthhdc" in resourcegroup="rg-candidate-4-21-h7c9rnh9n24x" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b55cb22c-9a55-40c5-b3c6-893ec03a9fc6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/b55cb22c-9a55-40c5-b3c6-893ec03a9fc6",
      "name": "b55cb22c-9a55-40c5-b3c6-893ec03a9fc6",
      "status": "Failed",
      "startTime": "2026-09-22T21:07:20.636049246Z",
      "endTime": "2026-09-22T21:36:27.958431734Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 2m58s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

failed to locate cilium chart: Get "<url>": dial tcp <ip>: connect: network is unreachable

19:50:05 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:159]: failed to install Cilium chart via Helm
Unexpected error:
    <*fmt.wrapError | 0xc0008581e0>: 
    failed to locate cilium chart: Get "https://helm.cilium.io/cilium-1.19.2.tgz": dial tcp [2606:50c0:8000::153]:443: connect: network is unreachable
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:159]: failed to install Cilium chart via Helm
Unexpected error:
    <*fmt.wrapError | 0xc0008581e0>: 
    failed to locate cilium chart: Get "https://helm.cilium.io/cilium-1.19.2.tgz": dial tcp [2606:50c0:8000::153]:443: connect: network is unreachable
    ...
occurred

19:50:05 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite integration/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000914940>: 
    failed to create HCP cluster ea-list7cctv9: failed waiting for cluster="ea-list7cctv9" in resourcegroup="ea-list-wc6fbxsd9rxx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3a3e500-1ba1-4cb7-8434-bc1aa96ee070
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3a3e500-1ba1-4cb7-8434-bc1aa96ee070",
      "name": "c3a3e500-1ba1-4cb7-8434-bc1aa96ee070",
      "status": "Failed",
      "startTime": "2026-09-22T21:22:56.593640254Z",
      "endTime": "2026-09-22T21:51:58.372098632Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 2m51s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000914940>: 
    failed to create HCP cluster ea-list7cctv9: failed waiting for cluster="ea-list7cctv9" in resourcegroup="ea-list-wc6fbxsd9rxx" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3a3e500-1ba1-4cb7-8434-bc1aa96ee070
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c3a3e500-1ba1-4cb7-8434-bc1aa96ee070",
      "name": "c3a3e500-1ba1-4cb7-8434-bc1aa96ee070",
      "status": "Failed",
      "startTime": "2026-09-22T21:22:56.593640254Z",
      "endTime": "2026-09-22T21:51:58.372098632Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 2m51s ago; hosted cluster degraded: UnavailableReplicas: ignition-server-proxy deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

16:24:26

2h 13m 13s

branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel

post-goodmerged PR

No additional run metadata captured

westus3

n/a

passed

n/a

13:11:15

2h 28m 25s

branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel

post-goodmerged PR

No additional run metadata captured

westus3

n/a

passed

n/a

Environment: PROD

Time (UTC)	Runtime	Job	Region	Failed at	Result	PR	Failed tests	Details
23:00:41	2h 6m 38s	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly post-goodmerged PR No additional run metadata captured	uksouth	e2e	failed	n/a	14	e2e (14) ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541 Unexpected error: <fmt.wrapError \| 0xc000a7ee40>: failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-tcgbnw" in resourcegroup="rg-np-version-upgrade-tcgbnw-hj2k4f59tdf7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tcgbnw-hj2k4f59tdf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tcgbnw/nodePools/npupgrade-4-20 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541 Unexpected error: <fmt.wrapError \| 0xc000a7ee40>: failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-tcgbnw" in resourcegroup="rg-np-version-upgrade-tcgbnw-hj2k4f59tdf7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tcgbnw-hj2k4f59tdf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tcgbnw/nodePools/npupgrade-4-20 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update Unexpected error: <exported.ResponseError \| 0xc002790330>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rnbkplrwl7s9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update Unexpected error: <exported.ResponseError \| 0xc002790330>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rnbkplrwl7s9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred cluster CA data must contain exactly one PEM block 23:00:41 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:202]: cluster CA data must contain exactly one PEM block Expected <[]uint8 \| len:6010, cap:6012>: ... to be empty fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:202]: cluster CA data must contain exactly one PEM block Expected <[]uint8 \| len:6010, cap:6012>: ... to be empty ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration Unexpected error: <exported.ResponseError \| 0xc002942ed0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-khkzzgtsqgjz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration Unexpected error: <exported.ResponseError \| 0xc002942ed0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-khkzzgtsqgjz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster Unexpected error: <fmt.wrapError \| 0xc000ef1980>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-dkldtr9wz7m4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/baff5654-9ec0-4bc5-a8a1-ffc26382827a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/baff5654-9ec0-4bc5-a8a1-ffc26382827a", "name": "baff5654-9ec0-4bc5-a8a1-ffc26382827a", "status": "Failed", "startTime": "2026-09-22T23:22:56.369741564Z", "endTime": "2026-09-22T23:42:02.437458738Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster Unexpected error: <fmt.wrapError \| 0xc000ef1980>: failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-dkldtr9wz7m4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/baff5654-9ec0-4bc5-a8a1-ffc26382827a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/baff5654-9ec0-4bc5-a8a1-ffc26382827a", "name": "baff5654-9ec0-4bc5-a8a1-ffc26382827a", "status": "Failed", "startTime": "2026-09-22T23:22:56.369741564Z", "endTime": "2026-09-22T23:42:02.437458738Z", "error": { "code": "InternalServerError", "message": "node pool creation did not complete before the deadline" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs Unexpected error: <exported.ResponseError \| 0xc0025133b0>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-cxzbgzk6bm77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs Unexpected error: <exported.ResponseError \| 0xc0025133b0>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-cxzbgzk6bm77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set Unexpected error: <exported.ResponseError \| 0xc001002060>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-cqnjd26wmjcd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set Unexpected error: <exported.ResponseError \| 0xc001002060>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-cqnjd26wmjcd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-6gs84k" for cluster "cluster-nightly-4-20-6gs84k" Unexpected error: <fmt.wrapError \| 0xc000ce4120>: failed to create NodePool np-6gs84k: failed starting nodepool creation "np-6gs84k" for cluster "cluster-nightly-4-20-6gs84k" in resourcegroup="rg-nightly-4-20-cv5qj2kl686k": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-cv5qj2kl686k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-6gs84k/nodePools/np-6gs84k -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-6gs84k" for cluster "cluster-nightly-4-20-6gs84k" Unexpected error: <fmt.wrapError \| 0xc000ce4120>: failed to create NodePool np-6gs84k: failed starting nodepool creation "np-6gs84k" for cluster "cluster-nightly-4-20-6gs84k" in resourcegroup="rg-nightly-4-20-cv5qj2kl686k": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-cv5qj2kl686k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-6gs84k/nodePools/np-6gs84k -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-13-024541 Unexpected error: <fmt.wrapError \| 0xc0010981a0>: failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-x28k9h" in resourcegroup="rg-np-skip-minor-x28k9h-x2cmwhr29p2s": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-x28k9h-x2cmwhr29p2s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-x28k9h/nodePools/nps-420-422 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-13-024541 Unexpected error: <fmt.wrapError \| 0xc0010981a0>: failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-x28k9h" in resourcegroup="rg-np-skip-minor-x28k9h-x2cmwhr29p2s": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-x28k9h-x2cmwhr29p2s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-x28k9h/nodePools/nps-420-422 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH Unexpected error: <exported.ResponseError \| 0xc00148e5d0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-7dfqps65dplq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH Unexpected error: <exported.ResponseError \| 0xc00148e5d0>: PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-7dfqps65dplq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-1-ctwh8htmlpcs/cluster-nightly-5-1-v48g4k should provision Unexpected error: <fmt.wrapError \| 0xc00150a700>: failed to create HCP cluster cluster-nightly-5-1-v48g4k: failed waiting for cluster="cluster-nightly-5-1-v48g4k" in resourcegroup="rg-nightly-5-1-ctwh8htmlpcs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/219d6b96-27bd-42d1-8102-d400db0e0bb5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/219d6b96-27bd-42d1-8102-d400db0e0bb5", "name": "219d6b96-27bd-42d1-8102-d400db0e0bb5", "status": "Failed", "startTime": "2026-09-22T23:38:52.209443302Z", "endTime": "2026-09-23T00:07:57.16268573Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-1-ctwh8htmlpcs/cluster-nightly-5-1-v48g4k should provision Unexpected error: <fmt.wrapError \| 0xc00150a700>: failed to create HCP cluster cluster-nightly-5-1-v48g4k: failed waiting for cluster="cluster-nightly-5-1-v48g4k" in resourcegroup="rg-nightly-5-1-ctwh8htmlpcs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/219d6b96-27bd-42d1-8102-d400db0e0bb5 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/219d6b96-27bd-42d1-8102-d400db0e0bb5", "name": "219d6b96-27bd-42d1-8102-d400db0e0bb5", "status": "Failed", "startTime": "2026-09-22T23:38:52.209443302Z", "endTime": "2026-09-23T00:07:57.16268573Z", "error": { "code": "InternalServerError", "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet" } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.22": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc00103f360>: failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-xqlnpq2wtgvl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-xqlnpq2wtgvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc00103f360>: failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-xqlnpq2wtgvl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-xqlnpq2wtgvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) 23:00:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s. Expected success, but got an error: <errors.joinError \| 0xc000fb5908>: VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-21-210852, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053" ... fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s. Expected success, but got an error: <errors.joinError \| 0xc000fb5908>: VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade) VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-21-210852, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053" ... ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541 Unexpected error: <fmt.wrapError \| 0xc001385da0>: failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-4tjrbb" resourcegroup="rg-np-dg-minor-4tjrbb-jd2b4bxb9q5x": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-4tjrbb-jd2b4bxb9q5x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-4tjrbb/nodePools/npdg-422-420 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541 Unexpected error: <fmt.wrapError \| 0xc001385da0>: failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-4tjrbb" resourcegroup="rg-np-dg-minor-4tjrbb-jd2b4bxb9q5x": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-4tjrbb-jd2b4bxb9q5x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-4tjrbb/nodePools/npdg-422-420 -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidRequestContent -------------------------------------------------------------------------------- { "error": { "code": "InvalidRequestContent", "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8", "target": "properties.version.id" } } -------------------------------------------------------------------------------- ... occurred
02:00:40	2h 9m 43s	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	uksouth	n/a	passed	n/a	0	n/a

Time (UTC)

Runtime

Job

Region

Failed at

Result

Failed tests

Details

23:00:41

2h 6m 38s

periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly

post-goodmerged PR

No additional run metadata captured

uksouth

e2e

failed

n/a

e2e (14)

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000a7ee40>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-tcgbnw" in resourcegroup="rg-np-version-upgrade-tcgbnw-hj2k4f59tdf7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tcgbnw-hj2k4f59tdf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tcgbnw/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000a7ee40>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-tcgbnw" in resourcegroup="rg-np-version-upgrade-tcgbnw-hj2k4f59tdf7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-tcgbnw-hj2k4f59tdf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-tcgbnw/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc002790330>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rnbkplrwl7s9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc002790330>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rnbkplrwl7s9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

cluster CA data must contain exactly one PEM block

23:00:41 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:202]: cluster CA data must contain exactly one PEM block
Expected
    <[]uint8 | len:6010, cap:6012>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:202]: cluster CA data must contain exactly one PEM block
Expected
    <[]uint8 | len:6010, cap:6012>: ...
to be empty

23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc002942ed0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-khkzzgtsqgjz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc002942ed0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-khkzzgtsqgjz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ef1980>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-dkldtr9wz7m4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/baff5654-9ec0-4bc5-a8a1-ffc26382827a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/baff5654-9ec0-4bc5-a8a1-ffc26382827a",
      "name": "baff5654-9ec0-4bc5-a8a1-ffc26382827a",
      "status": "Failed",
      "startTime": "2026-09-22T23:22:56.369741564Z",
      "endTime": "2026-09-22T23:42:02.437458738Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:156]: failed to create node pool np-1 for TLS endpoint cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ef1980>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-dkldtr9wz7m4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/baff5654-9ec0-4bc5-a8a1-ffc26382827a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/baff5654-9ec0-4bc5-a8a1-ffc26382827a",
      "name": "baff5654-9ec0-4bc5-a8a1-ffc26382827a",
      "status": "Failed",
      "startTime": "2026-09-22T23:22:56.369741564Z",
      "endTime": "2026-09-22T23:42:02.437458738Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

23:00:41 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc0025133b0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-cxzbgzk6bm77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc0025133b0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-cxzbgzk6bm77/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc001002060>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-cqnjd26wmjcd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc001002060>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-cqnjd26wmjcd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20 · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-6gs84k" for cluster "cluster-nightly-4-20-6gs84k"
Unexpected error:
    <*fmt.wrapError | 0xc000ce4120>: 
    failed to create NodePool np-6gs84k: failed starting nodepool creation "np-6gs84k" for cluster "cluster-nightly-4-20-6gs84k" in resourcegroup="rg-nightly-4-20-cv5qj2kl686k": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-cv5qj2kl686k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-6gs84k/nodePools/np-6gs84k
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-6gs84k" for cluster "cluster-nightly-4-20-6gs84k"
Unexpected error:
    <*fmt.wrapError | 0xc000ce4120>: 
    failed to create NodePool np-6gs84k: failed starting nodepool creation "np-6gs84k" for cluster "cluster-nightly-4-20-6gs84k" in resourcegroup="rg-nightly-4-20-cv5qj2kl686k": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-cv5qj2kl686k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-6gs84k/nodePools/np-6gs84k
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc0010981a0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-x28k9h" in resourcegroup="rg-np-skip-minor-x28k9h-x2cmwhr29p2s": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-x28k9h-x2cmwhr29p2s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-x28k9h/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc0010981a0>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-x28k9h" in resourcegroup="rg-np-skip-minor-x28k9h-x2cmwhr29p2s": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-x28k9h-x2cmwhr29p2s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-x28k9h/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

23:00:41 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc00148e5d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-7dfqps65dplq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc00148e5d0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-7dfqps65dplq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: InternalServerError; detail message [cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1 · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-1-ctwh8htmlpcs/cluster-nightly-5-1-v48g4k should provision
Unexpected error:
    <*fmt.wrapError | 0xc00150a700>: 
    failed to create HCP cluster cluster-nightly-5-1-v48g4k: failed waiting for cluster="cluster-nightly-5-1-v48g4k" in resourcegroup="rg-nightly-5-1-ctwh8htmlpcs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/219d6b96-27bd-42d1-8102-d400db0e0bb5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/219d6b96-27bd-42d1-8102-d400db0e0bb5",
      "name": "219d6b96-27bd-42d1-8102-d400db0e0bb5",
      "status": "Failed",
      "startTime": "2026-09-22T23:38:52.209443302Z",
      "endTime": "2026-09-23T00:07:57.16268573Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-1-ctwh8htmlpcs/cluster-nightly-5-1-v48g4k should provision
Unexpected error:
    <*fmt.wrapError | 0xc00150a700>: 
    failed to create HCP cluster cluster-nightly-5-1-v48g4k: failed waiting for cluster="cluster-nightly-5-1-v48g4k" in resourcegroup="rg-nightly-5-1-ctwh8htmlpcs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/219d6b96-27bd-42d1-8102-d400db0e0bb5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/219d6b96-27bd-42d1-8102-d400db0e0bb5",
      "name": "219d6b96-27bd-42d1-8102-d400db0e0bb5",
      "status": "Failed",
      "startTime": "2026-09-22T23:38:52.209443302Z",
      "endTime": "2026-09-23T00:07:57.16268573Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.22": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00103f360>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-xqlnpq2wtgvl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-xqlnpq2wtgvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00103f360>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-xqlnpq2wtgvl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-xqlnpq2wtgvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)

23:00:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc000fb5908>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-21-210852, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.006s.
Expected success, but got an error:
    <*errors.joinError | 0xc000fb5908>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-21-210852, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...

ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc001385da0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-4tjrbb" resourcegroup="rg-np-dg-minor-4tjrbb-jd2b4bxb9q5x": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-4tjrbb-jd2b4bxb9q5x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-4tjrbb/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc001385da0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-4tjrbb" resourcegroup="rg-np-dg-minor-4tjrbb-jd2b4bxb9q5x": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-4tjrbb-jd2b4bxb9q5x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-4tjrbb/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-13-024541\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

02:00:40

2h 9m 43s

periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel

post-goodmerged PR

No additional run metadata captured

uksouth

n/a

passed

n/a

Environment: STG

Time (UTC)	Runtime	Job	Region	Failed at	Result	PR	Failed tests	Details
23:00:41	15m 34s	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly post-goodmerged PR No additional run metadata captured	uksouth	e2e	failed	n/a	52	e2e (52) ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001022320>: failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-5c69lbjzcdrv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-5c69lbjzcdrv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001022320>: failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-5c69lbjzcdrv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-5c69lbjzcdrv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-9r6pb2" at install version 4.20.0-0.nightly-multi-2026-09-13-024541 Unexpected error: <fmt.wrapError \| 0xc000618600>: failed to create HCP cluster cp-ystream-upgrade-4-21-9r6pb2: failed starting cluster creation "cp-ystream-upgrade-4-21-9r6pb2" in resourcegroup="rg-cp-ystream-upgrade-4-21-8wrqg8zdkgps": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-8wrqg8zdkgps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-9r6pb2 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-9r6pb2" at install version 4.20.0-0.nightly-multi-2026-09-13-024541 Unexpected error: <fmt.wrapError \| 0xc000618600>: failed to create HCP cluster cp-ystream-upgrade-4-21-9r6pb2: failed starting cluster creation "cp-ystream-upgrade-4-21-9r6pb2" in resourcegroup="rg-cp-ystream-upgrade-4-21-8wrqg8zdkgps": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-8wrqg8zdkgps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-9r6pb2 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to list available HCP OpenShift versions and validate response content · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions Unexpected error: <exported.ResponseError \| 0xc001597a40>: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions Unexpected error: <exported.ResponseError \| 0xc001597a40>: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000f88140>: failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-8vpkwbcpkrmb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-8vpkwbcpkrmb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000f88140>: failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-8vpkwbcpkrmb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-8vpkwbcpkrmb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to delete an HCP cluster whose managed identities were deleted first · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc001782240>: failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-2xjmn786m79q": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-2xjmn786m79q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc001782240>: failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-2xjmn786m79q": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-2xjmn786m79q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-20-5jxwkvlcphv4/cluster-nightly-4-20-59hchb should provision Unexpected error: <fmt.wrapError \| 0xc0010903c0>: failed to create HCP cluster cluster-nightly-4-20-59hchb: failed starting cluster creation "cluster-nightly-4-20-59hchb" in resourcegroup="rg-nightly-4-20-5jxwkvlcphv4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-5jxwkvlcphv4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-59hchb -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-20-5jxwkvlcphv4/cluster-nightly-4-20-59hchb should provision Unexpected error: <fmt.wrapError \| 0xc0010903c0>: failed to create HCP cluster cluster-nightly-4-20-59hchb: failed starting cluster creation "cluster-nightly-4-20-59hchb" in resourcegroup="rg-nightly-4-20-5jxwkvlcphv4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-5jxwkvlcphv4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-59hchb -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-mtz6w6 at version 4.22.0-0.nightly-multi-2026-09-17-071442 Unexpected error: <fmt.wrapError \| 0xc000bf4220>: failed to create HCP cluster np-skip-minor-mtz6w6: failed starting cluster creation "np-skip-minor-mtz6w6" in resourcegroup="rg-np-skip-minor-mtz6w6-kplfq87nckfq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-mtz6w6-kplfq87nckfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-mtz6w6 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-mtz6w6 at version 4.22.0-0.nightly-multi-2026-09-17-071442 Unexpected error: <fmt.wrapError \| 0xc000bf4220>: failed to create HCP cluster np-skip-minor-mtz6w6: failed starting cluster creation "np-skip-minor-mtz6w6" in resourcegroup="rg-np-skip-minor-mtz6w6-kplfq87nckfq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-mtz6w6-kplfq87nckfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-mtz6w6 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000a562c0>: failed to create HCP cluster ea-listt8rz8v: failed starting cluster creation "ea-listt8rz8v" in resourcegroup="ea-list-bvk5zpjkh2nd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-bvk5zpjkh2nd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listt8rz8v -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000a562c0>: failed to create HCP cluster ea-listt8rz8v: failed starting cluster creation "ea-listt8rz8v" in resourcegroup="ea-list-bvk5zpjkh2nd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-bvk5zpjkh2nd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listt8rz8v -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-nn5gnm" at install version 4.22.0-0.nightly-multi-2026-09-17-071442 Unexpected error: <fmt.wrapError \| 0xc000b5a0e0>: failed to create HCP cluster cp-ystream-upgrade-5-0-nn5gnm: failed starting cluster creation "cp-ystream-upgrade-5-0-nn5gnm" in resourcegroup="rg-cp-ystream-upgrade-5-0-cn4pm599hjxc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-cn4pm599hjxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-nn5gnm -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-nn5gnm" at install version 4.22.0-0.nightly-multi-2026-09-17-071442 Unexpected error: <fmt.wrapError \| 0xc000b5a0e0>: failed to create HCP cluster cp-ystream-upgrade-5-0-nn5gnm: failed starting cluster creation "cp-ystream-upgrade-5-0-nn5gnm" in resourcegroup="rg-cp-ystream-upgrade-5-0-cn4pm599hjxc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-cn4pm599hjxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-nn5gnm -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.0 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-0-8kk449x5s6qc/cluster-nightly-5-0-bhqwr9 should provision Unexpected error: <fmt.wrapError \| 0xc000e4e280>: failed to create HCP cluster cluster-nightly-5-0-bhqwr9: failed starting cluster creation "cluster-nightly-5-0-bhqwr9" in resourcegroup="rg-nightly-5-0-8kk449x5s6qc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-8kk449x5s6qc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-0-bhqwr9 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-0-8kk449x5s6qc/cluster-nightly-5-0-bhqwr9 should provision Unexpected error: <fmt.wrapError \| 0xc000e4e280>: failed to create HCP cluster cluster-nightly-5-0-bhqwr9: failed starting cluster creation "cluster-nightly-5-0-bhqwr9" in resourcegroup="rg-nightly-5-0-8kk449x5s6qc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-8kk449x5s6qc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-0-bhqwr9 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc00101bd80>: failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-jnltm5nznwcw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-jnltm5nznwcw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc00101bd80>: failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-jnltm5nznwcw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-jnltm5nznwcw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000698d00>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-9ggtk5tbv4fl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-9ggtk5tbv4fl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000698d00>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-9ggtk5tbv4fl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-9ggtk5tbv4fl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc001592140>: failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-xc8jsrhsbqgs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-xc8jsrhsbqgs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc001592140>: failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-xc8jsrhsbqgs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-xc8jsrhsbqgs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc00011ba60>: failed to create HCP cluster np-dg-minor-hgh59k: failed starting cluster creation "np-dg-minor-hgh59k" in resourcegroup="rg-np-dg-minor-hgh59k-49x99ndztwrj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-hgh59k-49x99ndztwrj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-hgh59k -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc00011ba60>: failed to create HCP cluster np-dg-minor-hgh59k: failed starting cluster creation "np-dg-minor-hgh59k" in resourcegroup="rg-np-dg-minor-hgh59k-49x99ndztwrj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-hgh59k-49x99ndztwrj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-hgh59k -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc00096a120>: failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-dw22pxrnfnk6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-dw22pxrnfnk6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc00096a120>: failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-dw22pxrnfnk6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-dw22pxrnfnk6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000b90100>: failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-g86rpg6nvsxm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-g86rpg6nvsxm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc000b90100>: failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-g86rpg6nvsxm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-g86rpg6nvsxm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0xc001180240>: failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-hbfdw9rw98ps": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-hbfdw9rw98ps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0xc001180240>: failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-hbfdw9rw98ps": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-hbfdw9rw98ps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000c92540>: failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-dr6j2dvf9kv5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-dr6j2dvf9kv5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000c92540>: failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-dr6j2dvf9kv5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-dr6j2dvf9kv5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test Unexpected error: <fmt.wrapError \| 0xc000fa4120>: failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-x4n5jq88swcc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-x4n5jq88swcc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test Unexpected error: <fmt.wrapError \| 0xc000fa4120>: failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-x4n5jq88swcc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-x4n5jq88swcc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000618180>: failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-5g9nckwfbg5h": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-5g9nckwfbg5h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000618180>: failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-5g9nckwfbg5h": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-5g9nckwfbg5h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc000fd6100>: failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nnlw6xnxb26v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-nnlw6xnxb26v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc000fd6100>: failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nnlw6xnxb26v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-nnlw6xnxb26v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-sctgzn" at install version 4.22.0-0.nightly-multi-2026-09-17-071442 Unexpected error: <fmt.wrapError \| 0xc000d7a1a0>: failed to create HCP cluster cp-ystream-upgrade-4-23-sctgzn: failed starting cluster creation "cp-ystream-upgrade-4-23-sctgzn" in resourcegroup="rg-cp-ystream-upgrade-4-23-vdl85ff8lc86": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-vdl85ff8lc86/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-sctgzn -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-sctgzn" at install version 4.22.0-0.nightly-multi-2026-09-17-071442 Unexpected error: <fmt.wrapError \| 0xc000d7a1a0>: failed to create HCP cluster cp-ystream-upgrade-4-23-sctgzn: failed starting cluster creation "cp-ystream-upgrade-4-23-sctgzn" in resourcegroup="rg-cp-ystream-upgrade-4-23-vdl85ff8lc86": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-vdl85ff8lc86/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-sctgzn -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc0012a63e0>: failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-ms56qpvz7f4t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ms56qpvz7f4t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc0012a63e0>: failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-ms56qpvz7f4t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ms56qpvz7f4t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:103]: failed to begin creating HCP cluster "admin-cred-lifecycle-2r66p6" Unexpected error: <fmt.wrapError \| 0xc000abad00>: failed starting cluster creation "admin-cred-lifecycle-2r66p6" in resourcegroup="admin-credential-lifecycle-test-5fm57kncmfth": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-5fm57kncmfth/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-2r66p6 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:103]: failed to begin creating HCP cluster "admin-cred-lifecycle-2r66p6" Unexpected error: <fmt.wrapError \| 0xc000abad00>: failed starting cluster creation "admin-cred-lifecycle-2r66p6" in resourcegroup="admin-credential-lifecycle-test-5fm57kncmfth": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-5fm57kncmfth/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-2r66p6 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20 Unexpected error: <fmt.wrapError \| 0xc000d86180>: failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-9h8ggzx7ljbh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-9h8ggzx7ljbh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20 Unexpected error: <fmt.wrapError \| 0xc000d86180>: failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-9h8ggzx7ljbh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-9h8ggzx7ljbh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation Unexpected error: <exported.ResponseError \| 0xc001b18840>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-ntrplhzxjq86/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation Unexpected error: <exported.ResponseError \| 0xc001b18840>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-ntrplhzxjq86/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc0005ae660>: failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-8zk685kvqsgq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8zk685kvqsgq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc0005ae660>: failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-8zk685kvqsgq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8zk685kvqsgq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-1-7m29nrv4fdr6/cluster-nightly-5-1-scx85f should provision Unexpected error: <fmt.wrapError \| 0xc0013fc280>: failed to create HCP cluster cluster-nightly-5-1-scx85f: failed starting cluster creation "cluster-nightly-5-1-scx85f" in resourcegroup="rg-nightly-5-1-7m29nrv4fdr6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-7m29nrv4fdr6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-1-scx85f -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-1-7m29nrv4fdr6/cluster-nightly-5-1-scx85f should provision Unexpected error: <fmt.wrapError \| 0xc0013fc280>: failed to create HCP cluster cluster-nightly-5-1-scx85f: failed starting cluster creation "cluster-nightly-5-1-scx85f" in resourcegroup="rg-nightly-5-1-7m29nrv4fdr6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-7m29nrv4fdr6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-1-scx85f -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc001030240>: failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-42c2h57lg25f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-42c2h57lg25f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc001030240>: failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-42c2h57lg25f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-42c2h57lg25f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000682260>: failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-j585nqsnf67j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-j585nqsnf67j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000682260>: failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-j585nqsnf67j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-j585nqsnf67j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-c4n9pr with version 4.21.0-0.nightly-multi-2026-09-22-123437 Unexpected error: <fmt.wrapError \| 0xc000bb8780>: failed to create HCP cluster np-version-upgrade-cluster-c4n9pr: failed starting cluster creation "np-version-upgrade-cluster-c4n9pr" in resourcegroup="rg-np-version-upgrade-c4n9pr-lvc25kgsdccb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-c4n9pr-lvc25kgsdccb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-c4n9pr -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-c4n9pr with version 4.21.0-0.nightly-multi-2026-09-22-123437 Unexpected error: <fmt.wrapError \| 0xc000bb8780>: failed to create HCP cluster np-version-upgrade-cluster-c4n9pr: failed starting cluster creation "np-version-upgrade-cluster-c4n9pr" in resourcegroup="rg-np-version-upgrade-c4n9pr-lvc25kgsdccb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-c4n9pr-lvc25kgsdccb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-c4n9pr -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.22 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-22-r8dcl75vwsvm/cluster-nightly-4-22-rbw68x should provision Unexpected error: <fmt.wrapError \| 0xc000714200>: failed to create HCP cluster cluster-nightly-4-22-rbw68x: failed starting cluster creation "cluster-nightly-4-22-rbw68x" in resourcegroup="rg-nightly-4-22-r8dcl75vwsvm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-r8dcl75vwsvm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-22-rbw68x -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-22-r8dcl75vwsvm/cluster-nightly-4-22-rbw68x should provision Unexpected error: <fmt.wrapError \| 0xc000714200>: failed to create HCP cluster cluster-nightly-4-22-rbw68x: failed starting cluster creation "cluster-nightly-4-22-rbw68x" in resourcegroup="rg-nightly-4-22-r8dcl75vwsvm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-r8dcl75vwsvm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-22-rbw68x -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test Unexpected error: <fmt.wrapError \| 0xc00011b8a0>: failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-877p4644xp66": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-877p4644xp66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test Unexpected error: <fmt.wrapError \| 0xc00011b8a0>: failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-877p4644xp66": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-877p4644xp66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc000aec280>: failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-ggdtclvwvh49": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-ggdtclvwvh49/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc000aec280>: failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-ggdtclvwvh49": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-ggdtclvwvh49/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to cordon, drain, and uncordon a node in an HCP cluster · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc0001b3020>: failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-x64bd6kgw77d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-x64bd6kgw77d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc0001b3020>: failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-x64bd6kgw77d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-x64bd6kgw77d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000e98240>: failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-nkc9f66h7mrb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-nkc9f66h7mrb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000e98240>: failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-nkc9f66h7mrb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-nkc9f66h7mrb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc00140c260>: failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-bmczqgpdvhmg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-bmczqgpdvhmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc00140c260>: failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-bmczqgpdvhmg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-bmczqgpdvhmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0xc0010740e0>: failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-snrfdxp7lz6k": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-snrfdxp7lz6k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0xc0010740e0>: failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-snrfdxp7lz6k": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-snrfdxp7lz6k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000624460>: failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-72hqfj82n768": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-72hqfj82n768/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc000624460>: failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-72hqfj82n768": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-72hqfj82n768/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0xc00114e160>: failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-bv97895wknf4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-bv97895wknf4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0xc00114e160>: failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-bv97895wknf4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-bv97895wknf4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000c82a40>: failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-b6pmc9sdknkf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-b6pmc9sdknkf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000c82a40>: failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-b6pmc9sdknkf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-b6pmc9sdknkf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc0013160e0>: failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-rx9wkcqgtnhz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-rx9wkcqgtnhz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc0013160e0>: failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-rx9wkcqgtnhz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-rx9wkcqgtnhz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.23 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-23-bzpzlqbzxqvl/cluster-nightly-4-23-p7gkv7 should provision Unexpected error: <fmt.wrapError \| 0xc0009be140>: failed to create HCP cluster cluster-nightly-4-23-p7gkv7: failed starting cluster creation "cluster-nightly-4-23-p7gkv7" in resourcegroup="rg-nightly-4-23-bzpzlqbzxqvl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-bzpzlqbzxqvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-23-p7gkv7 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-23-bzpzlqbzxqvl/cluster-nightly-4-23-p7gkv7 should provision Unexpected error: <fmt.wrapError \| 0xc0009be140>: failed to create HCP cluster cluster-nightly-4-23-p7gkv7: failed starting cluster creation "cluster-nightly-4-23-p7gkv7" in resourcegroup="rg-nightly-4-23-bzpzlqbzxqvl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-bzpzlqbzxqvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-23-p7gkv7 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000886420>: failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-ps8hpf8pd6z9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-ps8hpf8pd6z9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000886420>: failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-ps8hpf8pd6z9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-ps8hpf8pd6z9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-frkjxj" at install version 5.0.0-0.nightly-multi-2026-09-21-210852 Unexpected error: <fmt.wrapError \| 0xc000bfe2e0>: failed to create HCP cluster cp-ystream-upgrade-5-1-frkjxj: failed starting cluster creation "cp-ystream-upgrade-5-1-frkjxj" in resourcegroup="rg-cp-ystream-upgrade-5-1-kcgn55zdv82f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-kcgn55zdv82f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-frkjxj -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-frkjxj" at install version 5.0.0-0.nightly-multi-2026-09-21-210852 Unexpected error: <fmt.wrapError \| 0xc000bfe2e0>: failed to create HCP cluster cp-ystream-upgrade-5-1-frkjxj: failed starting cluster creation "cp-ystream-upgrade-5-1-frkjxj" in resourcegroup="rg-cp-ystream-upgrade-5-1-kcgn55zdv82f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-kcgn55zdv82f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-frkjxj -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000744c40>: failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-2jnwvdzzh64d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-2jnwvdzzh64d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000744c40>: failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-2jnwvdzzh64d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-2jnwvdzzh64d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc001072300>: failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-gtbrw7nrxjz2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-gtbrw7nrxjz2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc001072300>: failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-gtbrw7nrxjz2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-gtbrw7nrxjz2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-kx2v6b" at install version 4.21.0-0.nightly-multi-2026-09-22-123437 Unexpected error: <fmt.wrapError \| 0xc000c83440>: failed to create HCP cluster cp-ystream-upgrade-4-22-kx2v6b: failed starting cluster creation "cp-ystream-upgrade-4-22-kx2v6b" in resourcegroup="rg-cp-ystream-upgrade-4-22-sqqmsdwsqbrs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-sqqmsdwsqbrs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-kx2v6b -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-kx2v6b" at install version 4.21.0-0.nightly-multi-2026-09-22-123437 Unexpected error: <fmt.wrapError \| 0xc000c83440>: failed to create HCP cluster cp-ystream-upgrade-4-22-kx2v6b: failed starting cluster creation "cp-ystream-upgrade-4-22-kx2v6b" in resourcegroup="rg-cp-ystream-upgrade-4-22-sqqmsdwsqbrs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-sqqmsdwsqbrs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-kx2v6b -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc001052180>: failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-khqv7gn8cz69": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-khqv7gn8cz69/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc001052180>: failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-khqv7gn8cz69": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-khqv7gn8cz69/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc00102c3a0>: failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-4qb52bqkpzjf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-4qb52bqkpzjf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc00102c3a0>: failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-4qb52bqkpzjf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-4qb52bqkpzjf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc0006784e0>: failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-l2fgd6zqtf9h": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l2fgd6zqtf9h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc0006784e0>: failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-l2fgd6zqtf9h": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l2fgd6zqtf9h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 23:00:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.21 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-21-v8sz4sgjpztm/cluster-nightly-4-21-9lgzcb should provision Unexpected error: <fmt.wrapError \| 0xc000cb22a0>: failed to create HCP cluster cluster-nightly-4-21-9lgzcb: failed starting cluster creation "cluster-nightly-4-21-9lgzcb" in resourcegroup="rg-nightly-4-21-v8sz4sgjpztm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-v8sz4sgjpztm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-21-9lgzcb -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-21-v8sz4sgjpztm/cluster-nightly-4-21-9lgzcb should provision Unexpected error: <fmt.wrapError \| 0xc000cb22a0>: failed to create HCP cluster cluster-nightly-4-21-9lgzcb: failed starting cluster creation "cluster-nightly-4-21-9lgzcb" in resourcegroup="rg-nightly-4-21-v8sz4sgjpztm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-v8sz4sgjpztm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-21-9lgzcb -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred
02:00:40	16m 10s	periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel post-goodmerged PR No additional run metadata captured	uksouth	e2e	failed	n/a	56	e2e (56) ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc000f300e0>: failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-rz6r289t49jf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-rz6r289t49jf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit Unexpected error: <fmt.wrapError \| 0xc000f300e0>: failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-rz6r289t49jf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-rz6r289t49jf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to cordon, drain, and uncordon a node in an HCP cluster · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000b821e0>: failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-474zgvs2k728": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-474zgvs2k728/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000b821e0>: failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-474zgvs2k728": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-474zgvs2k728/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-xbrtd7" with version 5.0.0-rc.2 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000b30480>: failed to create HCP cluster cluster-zstream-5-0-xbrtd7: failed starting cluster creation "cluster-zstream-5-0-xbrtd7" in resourcegroup="rg-zstream-upgrade-5-0-w6cvb247pkgq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-w6cvb247pkgq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xbrtd7 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-xbrtd7" with version 5.0.0-rc.2 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000b30480>: failed to create HCP cluster cluster-zstream-5-0-xbrtd7: failed starting cluster creation "cluster-zstream-5-0-xbrtd7" in resourcegroup="rg-zstream-upgrade-5-0-w6cvb247pkgq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-w6cvb247pkgq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xbrtd7 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc000bcca20>: failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-zrxpwlk9jdzp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-zrxpwlk9jdzp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test Unexpected error: <fmt.wrapError \| 0xc000bcca20>: failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-zrxpwlk9jdzp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-zrxpwlk9jdzp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000f1e0e0>: failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-8fc9fdnjqp2w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-8fc9fdnjqp2w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd Unexpected error: <fmt.wrapError \| 0xc000f1e0e0>: failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-8fc9fdnjqp2w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-8fc9fdnjqp2w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-h75t75dg5lvd/cluster-candidate-4-22-bc8pwv should provision Unexpected error: <fmt.wrapError \| 0xc00155ce80>: failed to create HCP cluster cluster-candidate-4-22-bc8pwv: failed starting cluster creation "cluster-candidate-4-22-bc8pwv" in resourcegroup="rg-candidate-4-22-h75t75dg5lvd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-h75t75dg5lvd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-22-bc8pwv -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-h75t75dg5lvd/cluster-candidate-4-22-bc8pwv should provision Unexpected error: <fmt.wrapError \| 0xc00155ce80>: failed to create HCP cluster cluster-candidate-4-22-bc8pwv: failed starting cluster creation "cluster-candidate-4-22-bc8pwv" in resourcegroup="rg-candidate-4-22-h75t75dg5lvd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-h75t75dg5lvd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-22-bc8pwv -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-5hgv94" with version 4.22.13 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000c6a200>: failed to create HCP cluster cluster-zstream-4-22-5hgv94: failed starting cluster creation "cluster-zstream-4-22-5hgv94" in resourcegroup="rg-zstream-upgrade-4-22-rg8drsghclvc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-rg8drsghclvc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-5hgv94 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-5hgv94" with version 4.22.13 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000c6a200>: failed to create HCP cluster cluster-zstream-4-22-5hgv94: failed starting cluster creation "cluster-zstream-4-22-5hgv94" in resourcegroup="rg-zstream-upgrade-4-22-rg8drsghclvc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-rg8drsghclvc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-5hgv94 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test Unexpected error: <fmt.wrapError \| 0xc0002a67a0>: failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-jdv6zjq9vb9z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-jdv6zjq9vb9z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test Unexpected error: <fmt.wrapError \| 0xc0002a67a0>: failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-jdv6zjq9vb9z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-jdv6zjq9vb9z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000ba2aa0>: failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-mgkv5vcnbk4t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-mgkv5vcnbk4t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc000ba2aa0>: failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-mgkv5vcnbk4t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-mgkv5vcnbk4t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000b89040>: failed to create HCP cluster ea-listnksgg8: failed starting cluster creation "ea-listnksgg8" in resourcegroup="ea-list-tx57g2bcffrp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-tx57g2bcffrp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listnksgg8 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test Unexpected error: <fmt.wrapError \| 0xc000b89040>: failed to create HCP cluster ea-listnksgg8: failed starting cluster creation "ea-listnksgg8" in resourcegroup="ea-list-tx57g2bcffrp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-tx57g2bcffrp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listnksgg8 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-zd2hv5 at version 4.22.14 Unexpected error: <fmt.wrapError \| 0xc001018dc0>: failed to create HCP cluster np-skip-minor-zd2hv5: failed starting cluster creation "np-skip-minor-zd2hv5" in resourcegroup="rg-np-skip-minor-zd2hv5-xg6gmd9sxw8t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-zd2hv5-xg6gmd9sxw8t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-zd2hv5 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-zd2hv5 at version 4.22.14 Unexpected error: <fmt.wrapError \| 0xc001018dc0>: failed to create HCP cluster np-skip-minor-zd2hv5: failed starting cluster creation "np-skip-minor-zd2hv5" in resourcegroup="rg-np-skip-minor-zd2hv5-xg6gmd9sxw8t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-zd2hv5-xg6gmd9sxw8t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-zd2hv5 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000f0e200>: failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-2w6r47m2vzbz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-2w6r47m2vzbz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000f0e200>: failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-2w6r47m2vzbz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-2w6r47m2vzbz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000640360>: failed to create HCP cluster np-downgrade-k7lqv2: failed starting cluster creation "np-downgrade-k7lqv2" in resourcegroup="rg-np-downgrade-k7lqv2-wq4hwbkxc846": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-k7lqv2-wq4hwbkxc846/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-k7lqv2 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000640360>: failed to create HCP cluster np-downgrade-k7lqv2: failed starting cluster creation "np-downgrade-k7lqv2" in resourcegroup="rg-np-downgrade-k7lqv2-wq4hwbkxc846": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-k7lqv2-wq4hwbkxc846/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-k7lqv2 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000707e00>: failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-pg4bl4582c6x": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-pg4bl4582c6x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster Unexpected error: <fmt.wrapError \| 0xc000707e00>: failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-pg4bl4582c6x": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-pg4bl4582c6x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-6ltghn4pvj58/cluster-candidate-4-21-s5gc4g should provision Unexpected error: <fmt.wrapError \| 0xc00072aea0>: failed to create HCP cluster cluster-candidate-4-21-s5gc4g: failed starting cluster creation "cluster-candidate-4-21-s5gc4g" in resourcegroup="rg-candidate-4-21-6ltghn4pvj58": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-6ltghn4pvj58/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-21-s5gc4g -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-6ltghn4pvj58/cluster-candidate-4-21-s5gc4g should provision Unexpected error: <fmt.wrapError \| 0xc00072aea0>: failed to create HCP cluster cluster-candidate-4-21-s5gc4g: failed starting cluster creation "cluster-candidate-4-21-s5gc4g" in resourcegroup="rg-candidate-4-21-6ltghn4pvj58": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-6ltghn4pvj58/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-21-s5gc4g -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc000eacda0>: failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-4wwnsq5nbvwk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-4wwnsq5nbvwk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints Unexpected error: <fmt.wrapError \| 0xc000eacda0>: failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-4wwnsq5nbvwk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-4wwnsq5nbvwk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc000c242a0>: failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-jh5p965r9h5p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jh5p965r9h5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster Unexpected error: <fmt.wrapError \| 0xc000c242a0>: failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-jh5p965r9h5p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jh5p965r9h5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc0009f61e0>: failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-plvwdtqxskcm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-plvwdtqxskcm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster Unexpected error: <fmt.wrapError \| 0xc0009f61e0>: failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-plvwdtqxskcm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-plvwdtqxskcm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-hrgb26 with version 4.21.34 Unexpected error: <fmt.wrapError \| 0xc000f8a340>: failed to create HCP cluster np-version-upgrade-cluster-hrgb26: failed starting cluster creation "np-version-upgrade-cluster-hrgb26" in resourcegroup="rg-np-version-upgrade-hrgb26-88r5lslmqpf7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-hrgb26-88r5lslmqpf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-hrgb26 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-hrgb26 with version 4.21.34 Unexpected error: <fmt.wrapError \| 0xc000f8a340>: failed to create HCP cluster np-version-upgrade-cluster-hrgb26: failed starting cluster creation "np-version-upgrade-cluster-hrgb26" in resourcegroup="rg-np-version-upgrade-hrgb26-88r5lslmqpf7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-hrgb26-88r5lslmqpf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-hrgb26 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000f101e0>: failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-8m2tcg7xkh2f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-8m2tcg7xkh2f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000f101e0>: failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-8m2tcg7xkh2f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-8m2tcg7xkh2f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000d920c0>: failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-dh726mjb6hfc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-dh726mjb6hfc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS Unexpected error: <fmt.wrapError \| 0xc000d920c0>: failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-dh726mjb6hfc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-dh726mjb6hfc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test Unexpected error: <fmt.wrapError \| 0xc00083e4e0>: failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-mqdcg88p65p5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-mqdcg88p65p5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test Unexpected error: <fmt.wrapError \| 0xc00083e4e0>: failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-mqdcg88p65p5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-mqdcg88p65p5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000da21c0>: failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-wzckxjx8ln9d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-wzckxjx8ln9d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test Unexpected error: <fmt.wrapError \| 0xc000da21c0>: failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-wzckxjx8ln9d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-wzckxjx8ln9d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-plbg5s" with version 4.20.37 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0014c45e0>: failed to create HCP cluster cluster-zstream-4-20-plbg5s: failed starting cluster creation "cluster-zstream-4-20-plbg5s" in resourcegroup="rg-zstream-upgrade-4-20-5dctr2v29hxm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-5dctr2v29hxm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-plbg5s -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-plbg5s" with version 4.20.37 on candidate channel Unexpected error: <fmt.wrapError \| 0xc0014c45e0>: failed to create HCP cluster cluster-zstream-4-20-plbg5s: failed starting cluster creation "cluster-zstream-4-20-plbg5s" in resourcegroup="rg-zstream-upgrade-4-20-5dctr2v29hxm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-5dctr2v29hxm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-plbg5s -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000726720>: failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-6mvbgcknfnql": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6mvbgcknfnql/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings Unexpected error: <fmt.wrapError \| 0xc000726720>: failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-6mvbgcknfnql": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6mvbgcknfnql/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20 Unexpected error: <fmt.wrapError \| 0xc0010ca2a0>: failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-wjdnbdcz7v5v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-wjdnbdcz7v5v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20 Unexpected error: <fmt.wrapError \| 0xc0010ca2a0>: failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-wjdnbdcz7v5v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-wjdnbdcz7v5v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to delete an HCP cluster whose managed identities were deleted first · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000f341e0>: failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-k4jvltkd7tqn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-k4jvltkd7tqn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000f341e0>: failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-k4jvltkd7tqn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-k4jvltkd7tqn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc0006f8420>: failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-chbpbwgpb8pr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-chbpbwgpb8pr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128" Unexpected error: <fmt.wrapError \| 0xc0006f8420>: failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-chbpbwgpb8pr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-chbpbwgpb8pr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-xvzwmw with version 4.21.34 Unexpected error: <fmt.wrapError \| 0xc000f141a0>: failed to create HCP cluster np-version-upgrade-cluster-xvzwmw: failed starting cluster creation "np-version-upgrade-cluster-xvzwmw" in resourcegroup="rg-np-version-upgrade-xvzwmw-bxjhmmlcjtxn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-xvzwmw-bxjhmmlcjtxn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-xvzwmw -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-xvzwmw with version 4.21.34 Unexpected error: <fmt.wrapError \| 0xc000f141a0>: failed to create HCP cluster np-version-upgrade-cluster-xvzwmw: failed starting cluster creation "np-version-upgrade-cluster-xvzwmw" in resourcegroup="rg-np-version-upgrade-xvzwmw-bxjhmmlcjtxn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-xvzwmw-bxjhmmlcjtxn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-xvzwmw -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000b48360>: failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-84h94kdhhq2z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-84h94kdhhq2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault Unexpected error: <fmt.wrapError \| 0xc000b48360>: failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-84h94kdhhq2z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-84h94kdhhq2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001000160>: failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-lbjqxnvvbh9l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-lbjqxnvvbh9l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc001000160>: failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-lbjqxnvvbh9l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-lbjqxnvvbh9l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0xc000d26300>: failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-kxlbk5nhx7dg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-kxlbk5nhx7dg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI Unexpected error: <fmt.wrapError \| 0xc000d26300>: failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-kxlbk5nhx7dg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-kxlbk5nhx7dg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc000aaa160>: failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-ww5k9bfrs47z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-ww5k9bfrs47z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk Unexpected error: <fmt.wrapError \| 0xc000aaa160>: failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-ww5k9bfrs47z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-ww5k9bfrs47z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc002880260>: failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-98wggrdxqbzz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-98wggrdxqbzz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster Unexpected error: <fmt.wrapError \| 0xc002880260>: failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-98wggrdxqbzz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-98wggrdxqbzz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc00103c2a0>: failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-lhpbhxvtfvqm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-lhpbhxvtfvqm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview Unexpected error: <fmt.wrapError \| 0xc00103c2a0>: failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-lhpbhxvtfvqm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-lhpbhxvtfvqm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-j8stln" at install version 4.20 Unexpected error: <fmt.wrapError \| 0xc001092140>: failed to create HCP cluster cp-ystream-upgrade-4-21-j8stln: failed starting cluster creation "cp-ystream-upgrade-4-21-j8stln" in resourcegroup="rg-cp-ystream-upgrade-4-21-k6fzpk2b68m4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-k6fzpk2b68m4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-j8stln -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-j8stln" at install version 4.20 Unexpected error: <fmt.wrapError \| 0xc001092140>: failed to create HCP cluster cp-ystream-upgrade-4-21-j8stln: failed starting cluster creation "cp-ystream-upgrade-4-21-j8stln" in resourcegroup="rg-cp-ystream-upgrade-4-21-k6fzpk2b68m4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-k6fzpk2b68m4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-j8stln -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc00118a260>: failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-q464fsngnb2z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-q464fsngnb2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling Unexpected error: <fmt.wrapError \| 0xc00118a260>: failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-q464fsngnb2z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-q464fsngnb2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-mckpkv Unexpected error: <fmt.wrapError \| 0xc000860720>: failed to create HCP cluster np-noedge-mckpkv: failed starting cluster creation "np-noedge-mckpkv" in resourcegroup="rg-np-noedge-mckpkv-chk6hnblbkth": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-mckpkv-chk6hnblbkth/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-mckpkv -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-mckpkv Unexpected error: <fmt.wrapError \| 0xc000860720>: failed to create HCP cluster np-noedge-mckpkv: failed starting cluster creation "np-noedge-mckpkv" in resourcegroup="rg-np-noedge-mckpkv-chk6hnblbkth": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-mckpkv-chk6hnblbkth/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-mckpkv -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation Unexpected error: <exported.ResponseError \| 0xc0010303c0>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-6t49grdzfqmk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation Unexpected error: <exported.ResponseError \| 0xc0010303c0>: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-6t49grdzfqmk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc001012360>: failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-d5hfbw5gmtt5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-d5hfbw5gmtt5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test Unexpected error: <fmt.wrapError \| 0xc001012360>: failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-d5hfbw5gmtt5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-d5hfbw5gmtt5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0xc00062c5e0>: failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-lczl6v6nrz6w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-lczl6v6nrz6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs Unexpected error: <fmt.wrapError \| 0xc00062c5e0>: failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-lczl6v6nrz6w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-lczl6v6nrz6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc00092c680>: failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-czpph9wzgls4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-czpph9wzgls4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors Unexpected error: <fmt.wrapError \| 0xc00092c680>: failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-czpph9wzgls4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-czpph9wzgls4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to list available HCP OpenShift versions and validate response content · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions Unexpected error: <exported.ResponseError \| 0xc001757c20>: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions Unexpected error: <exported.ResponseError \| 0xc001757c20>: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000b1af80>: failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-v2bhlsm92ksh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-v2bhlsm92ksh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS Unexpected error: <fmt.wrapError \| 0xc000b1af80>: failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-v2bhlsm92ksh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-v2bhlsm92ksh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-cxl549" at install version 4.22 Unexpected error: <fmt.wrapError \| 0xc0011c8240>: failed to create HCP cluster cp-ystream-upgrade-5-0-cxl549: failed starting cluster creation "cp-ystream-upgrade-5-0-cxl549" in resourcegroup="rg-cp-ystream-upgrade-5-0-klrb5nkdm6fv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-klrb5nkdm6fv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-cxl549 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-cxl549" at install version 4.22 Unexpected error: <fmt.wrapError \| 0xc0011c8240>: failed to create HCP cluster cp-ystream-upgrade-5-0-cxl549: failed starting cluster creation "cp-ystream-upgrade-5-0-cxl549" in resourcegroup="rg-cp-ystream-upgrade-5-0-klrb5nkdm6fv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-klrb5nkdm6fv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-cxl549 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-c2r948" with version 4.21.33 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000ed43c0>: failed to create HCP cluster cluster-zstream-4-21-c2r948: failed starting cluster creation "cluster-zstream-4-21-c2r948" in resourcegroup="rg-zstream-upgrade-4-21-swxb5fxkxx4z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-swxb5fxkxx4z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-c2r948 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-c2r948" with version 4.21.33 on candidate channel Unexpected error: <fmt.wrapError \| 0xc000ed43c0>: failed to create HCP cluster cluster-zstream-4-21-c2r948: failed starting cluster creation "cluster-zstream-4-21-c2r948" in resourcegroup="rg-zstream-upgrade-4-21-swxb5fxkxx4z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-swxb5fxkxx4z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-c2r948 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000fde240>: failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-qj86j7vf296r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-qj86j7vf296r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster" Unexpected error: <fmt.wrapError \| 0xc000fde240>: failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-qj86j7vf296r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-qj86j7vf296r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred failed to create HCP cluster <cluster>: failed starting cluster creation "<cluster>" in resourcegroup="<resource-group>": Put "<url>": stream error: stream ID 37; CANCEL; received from peer 02:00:40 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-6bkkt2 with version 4.20.38 Unexpected error: <fmt.wrapError \| 0xc0013940e0>: failed to create HCP cluster np-version-upgrade-cluster-6bkkt2: failed starting cluster creation "np-version-upgrade-cluster-6bkkt2" in resourcegroup="rg-np-version-upgrade-6bkkt2-m96mr2dxx5b8": Put "https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-6bkkt2-m96mr2dxx5b8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-6bkkt2?api-version=2024-06-10-preview": stream error: stream ID 37; CANCEL; received from peer ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-6bkkt2 with version 4.20.38 Unexpected error: <fmt.wrapError \| 0xc0013940e0>: failed to create HCP cluster np-version-upgrade-cluster-6bkkt2: failed starting cluster creation "np-version-upgrade-cluster-6bkkt2" in resourcegroup="rg-np-version-upgrade-6bkkt2-m96mr2dxx5b8": Put "https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-6bkkt2-m96mr2dxx5b8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-6bkkt2?api-version=2024-06-10-preview": stream error: stream ID 37; CANCEL; received from peer ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc00113a320>: failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-kbl8bmnn4b4w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-kbl8bmnn4b4w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress Unexpected error: <fmt.wrapError \| 0xc00113a320>: failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-kbl8bmnn4b4w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-kbl8bmnn4b4w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-g28hnvmc7v6c/cluster-candidate-4-20-762dl6 should provision Unexpected error: <fmt.wrapError \| 0xc000ca4160>: failed to create HCP cluster cluster-candidate-4-20-762dl6: failed starting cluster creation "cluster-candidate-4-20-762dl6" in resourcegroup="rg-candidate-4-20-g28hnvmc7v6c": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-g28hnvmc7v6c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-20-762dl6 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-g28hnvmc7v6c/cluster-candidate-4-20-762dl6 should provision Unexpected error: <fmt.wrapError \| 0xc000ca4160>: failed to create HCP cluster cluster-candidate-4-20-762dl6: failed starting cluster creation "cluster-candidate-4-20-762dl6" in resourcegroup="rg-candidate-4-20-g28hnvmc7v6c": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-g28hnvmc7v6c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-20-762dl6 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000da2400>: failed to create HCP cluster np-dg-minor-qh8fbr: failed starting cluster creation "np-dg-minor-qh8fbr" in resourcegroup="rg-np-dg-minor-qh8fbr-m5zhfcgf52j7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-qh8fbr-m5zhfcgf52j7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-qh8fbr -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster Unexpected error: <fmt.wrapError \| 0xc000da2400>: failed to create HCP cluster np-dg-minor-qh8fbr: failed starting cluster creation "np-dg-minor-qh8fbr" in resourcegroup="rg-np-dg-minor-qh8fbr-m5zhfcgf52j7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-qh8fbr-m5zhfcgf52j7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-qh8fbr -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-h79k8k" at install version 4.21 Unexpected error: <fmt.wrapError \| 0xc0014d33a0>: failed to create HCP cluster cp-ystream-upgrade-4-22-h79k8k: failed starting cluster creation "cp-ystream-upgrade-4-22-h79k8k" in resourcegroup="rg-cp-ystream-upgrade-4-22-w48jfg4cknrl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-w48jfg4cknrl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-h79k8k -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-h79k8k" at install version 4.21 Unexpected error: <fmt.wrapError \| 0xc0014d33a0>: failed to create HCP cluster cp-ystream-upgrade-4-22-h79k8k: failed starting cluster creation "cp-ystream-upgrade-4-22-h79k8k" in resourcegroup="rg-cp-ystream-upgrade-4-22-w48jfg4cknrl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-w48jfg4cknrl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-h79k8k -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-pxrfzhr4lgdl/cluster-candidate-5-0-gsk4z7 should provision Unexpected error: <fmt.wrapError \| 0xc00097ccc0>: failed to create HCP cluster cluster-candidate-5-0-gsk4z7: failed starting cluster creation "cluster-candidate-5-0-gsk4z7" in resourcegroup="rg-candidate-5-0-pxrfzhr4lgdl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-pxrfzhr4lgdl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-gsk4z7 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-pxrfzhr4lgdl/cluster-candidate-5-0-gsk4z7 should provision Unexpected error: <fmt.wrapError \| 0xc00097ccc0>: failed to create HCP cluster cluster-candidate-5-0-gsk4z7: failed starting cluster creation "cluster-candidate-5-0-gsk4z7" in resourcegroup="rg-candidate-5-0-pxrfzhr4lgdl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-pxrfzhr4lgdl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-gsk4z7 -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:105]: failed to begin creating HCP cluster "admin-cred-lifecycle-q52tkz" Unexpected error: <fmt.wrapError \| 0xc001098400>: failed starting cluster creation "admin-cred-lifecycle-q52tkz" in resourcegroup="admin-credential-lifecycle-test-ccbpf2vrpk7m": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-ccbpf2vrpk7m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-q52tkz -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:105]: failed to begin creating HCP cluster "admin-cred-lifecycle-q52tkz" Unexpected error: <fmt.wrapError \| 0xc001098400>: failed starting cluster creation "admin-cred-lifecycle-q52tkz" in resourcegroup="admin-credential-lifecycle-test-ccbpf2vrpk7m": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-ccbpf2vrpk7m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-q52tkz -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000a6e540>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-lq7mxkdms4nx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-lq7mxkdms4nx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled Unexpected error: <fmt.wrapError \| 0xc000a6e540>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-lq7mxkdms4nx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-lq7mxkdms4nx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift 02:00:40 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite stage/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0xc001956280>: failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-xfb8qbsjtxct": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-xfb8qbsjtxct/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster Unexpected error: <fmt.wrapError \| 0xc001956280>: failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-xfb8qbsjtxct": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-xfb8qbsjtxct/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster -------------------------------------------------------------------------------- RESPONSE 502: 502 Bad Gateway ERROR CODE: BadGatewayConnection -------------------------------------------------------------------------------- { "error": { "code": "BadGatewayConnection", "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request." } } -------------------------------------------------------------------------------- ... occurred

Time (UTC)

Runtime

Job

Region

Failed at

Result

Failed tests

Details

23:00:41

15m 34s

periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly

post-goodmerged PR

No additional run metadata captured

uksouth

e2e

failed

n/a

e2e (52)

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001022320>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-5c69lbjzcdrv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-5c69lbjzcdrv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001022320>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-5c69lbjzcdrv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-5c69lbjzcdrv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-9r6pb2" at install version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000618600>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-9r6pb2: failed starting cluster creation "cp-ystream-upgrade-4-21-9r6pb2" in resourcegroup="rg-cp-ystream-upgrade-4-21-8wrqg8zdkgps": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-8wrqg8zdkgps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-9r6pb2
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-9r6pb2" at install version 4.20.0-0.nightly-multi-2026-09-13-024541
Unexpected error:
    <*fmt.wrapError | 0xc000618600>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-9r6pb2: failed starting cluster creation "cp-ystream-upgrade-4-21-9r6pb2" in resourcegroup="rg-cp-ystream-upgrade-4-21-8wrqg8zdkgps": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-8wrqg8zdkgps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-9r6pb2
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to list available HCP OpenShift versions and validate response content · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc001597a40>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc001597a40>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000f88140>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-8vpkwbcpkrmb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-8vpkwbcpkrmb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000f88140>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-8vpkwbcpkrmb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-8vpkwbcpkrmb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to delete an HCP cluster whose managed identities were deleted first · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc001782240>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-2xjmn786m79q": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-2xjmn786m79q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc001782240>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-2xjmn786m79q": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-2xjmn786m79q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-20-5jxwkvlcphv4/cluster-nightly-4-20-59hchb should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010903c0>: 
    failed to create HCP cluster cluster-nightly-4-20-59hchb: failed starting cluster creation "cluster-nightly-4-20-59hchb" in resourcegroup="rg-nightly-4-20-5jxwkvlcphv4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-5jxwkvlcphv4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-59hchb
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-20-5jxwkvlcphv4/cluster-nightly-4-20-59hchb should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010903c0>: 
    failed to create HCP cluster cluster-nightly-4-20-59hchb: failed starting cluster creation "cluster-nightly-4-20-59hchb" in resourcegroup="rg-nightly-4-20-5jxwkvlcphv4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-5jxwkvlcphv4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-59hchb
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-mtz6w6 at version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc000bf4220>: 
    failed to create HCP cluster np-skip-minor-mtz6w6: failed starting cluster creation "np-skip-minor-mtz6w6" in resourcegroup="rg-np-skip-minor-mtz6w6-kplfq87nckfq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-mtz6w6-kplfq87nckfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-mtz6w6
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-mtz6w6 at version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc000bf4220>: 
    failed to create HCP cluster np-skip-minor-mtz6w6: failed starting cluster creation "np-skip-minor-mtz6w6" in resourcegroup="rg-np-skip-minor-mtz6w6-kplfq87nckfq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-mtz6w6-kplfq87nckfq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-mtz6w6
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000a562c0>: 
    failed to create HCP cluster ea-listt8rz8v: failed starting cluster creation "ea-listt8rz8v" in resourcegroup="ea-list-bvk5zpjkh2nd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-bvk5zpjkh2nd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listt8rz8v
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000a562c0>: 
    failed to create HCP cluster ea-listt8rz8v: failed starting cluster creation "ea-listt8rz8v" in resourcegroup="ea-list-bvk5zpjkh2nd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-bvk5zpjkh2nd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listt8rz8v
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-nn5gnm" at install version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc000b5a0e0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-nn5gnm: failed starting cluster creation "cp-ystream-upgrade-5-0-nn5gnm" in resourcegroup="rg-cp-ystream-upgrade-5-0-cn4pm599hjxc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-cn4pm599hjxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-nn5gnm
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-nn5gnm" at install version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc000b5a0e0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-nn5gnm: failed starting cluster creation "cp-ystream-upgrade-5-0-nn5gnm" in resourcegroup="rg-cp-ystream-upgrade-5-0-cn4pm599hjxc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-cn4pm599hjxc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-nn5gnm
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.0 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-0-8kk449x5s6qc/cluster-nightly-5-0-bhqwr9 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000e4e280>: 
    failed to create HCP cluster cluster-nightly-5-0-bhqwr9: failed starting cluster creation "cluster-nightly-5-0-bhqwr9" in resourcegroup="rg-nightly-5-0-8kk449x5s6qc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-8kk449x5s6qc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-0-bhqwr9
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-0-8kk449x5s6qc/cluster-nightly-5-0-bhqwr9 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000e4e280>: 
    failed to create HCP cluster cluster-nightly-5-0-bhqwr9: failed starting cluster creation "cluster-nightly-5-0-bhqwr9" in resourcegroup="rg-nightly-5-0-8kk449x5s6qc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-8kk449x5s6qc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-0-bhqwr9
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00101bd80>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-jnltm5nznwcw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-jnltm5nznwcw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00101bd80>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-jnltm5nznwcw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-jnltm5nznwcw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000698d00>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-9ggtk5tbv4fl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-9ggtk5tbv4fl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000698d00>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-9ggtk5tbv4fl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-9ggtk5tbv4fl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc001592140>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-xc8jsrhsbqgs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-xc8jsrhsbqgs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc001592140>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-xc8jsrhsbqgs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-xc8jsrhsbqgs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc00011ba60>: 
    failed to create HCP cluster np-dg-minor-hgh59k: failed starting cluster creation "np-dg-minor-hgh59k" in resourcegroup="rg-np-dg-minor-hgh59k-49x99ndztwrj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-hgh59k-49x99ndztwrj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-hgh59k
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc00011ba60>: 
    failed to create HCP cluster np-dg-minor-hgh59k: failed starting cluster creation "np-dg-minor-hgh59k" in resourcegroup="rg-np-dg-minor-hgh59k-49x99ndztwrj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-hgh59k-49x99ndztwrj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-hgh59k
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00096a120>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-dw22pxrnfnk6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-dw22pxrnfnk6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00096a120>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-dw22pxrnfnk6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-dw22pxrnfnk6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000b90100>: 
    failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-g86rpg6nvsxm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-g86rpg6nvsxm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000b90100>: 
    failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-g86rpg6nvsxm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-g86rpg6nvsxm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001180240>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-hbfdw9rw98ps": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-hbfdw9rw98ps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001180240>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-hbfdw9rw98ps": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-hbfdw9rw98ps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c92540>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-dr6j2dvf9kv5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-dr6j2dvf9kv5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c92540>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-dr6j2dvf9kv5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-dr6j2dvf9kv5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000fa4120>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-x4n5jq88swcc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-x4n5jq88swcc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000fa4120>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-x4n5jq88swcc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-x4n5jq88swcc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000618180>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-5g9nckwfbg5h": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-5g9nckwfbg5h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000618180>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-5g9nckwfbg5h": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-5g9nckwfbg5h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc000fd6100>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nnlw6xnxb26v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-nnlw6xnxb26v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc000fd6100>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nnlw6xnxb26v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-nnlw6xnxb26v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-sctgzn" at install version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc000d7a1a0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-sctgzn: failed starting cluster creation "cp-ystream-upgrade-4-23-sctgzn" in resourcegroup="rg-cp-ystream-upgrade-4-23-vdl85ff8lc86": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-vdl85ff8lc86/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-sctgzn
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-sctgzn" at install version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc000d7a1a0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-sctgzn: failed starting cluster creation "cp-ystream-upgrade-4-23-sctgzn" in resourcegroup="rg-cp-ystream-upgrade-4-23-vdl85ff8lc86": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-vdl85ff8lc86/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-sctgzn
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0012a63e0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-ms56qpvz7f4t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ms56qpvz7f4t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0012a63e0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-ms56qpvz7f4t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-ms56qpvz7f4t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:103]: failed to begin creating HCP cluster "admin-cred-lifecycle-2r66p6"
Unexpected error:
    <*fmt.wrapError | 0xc000abad00>: 
    failed starting cluster creation "admin-cred-lifecycle-2r66p6" in resourcegroup="admin-credential-lifecycle-test-5fm57kncmfth": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-5fm57kncmfth/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-2r66p6
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:103]: failed to begin creating HCP cluster "admin-cred-lifecycle-2r66p6"
Unexpected error:
    <*fmt.wrapError | 0xc000abad00>: 
    failed starting cluster creation "admin-cred-lifecycle-2r66p6" in resourcegroup="admin-credential-lifecycle-test-5fm57kncmfth": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-5fm57kncmfth/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-2r66p6
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.20 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000d86180>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-9h8ggzx7ljbh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-9h8ggzx7ljbh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000d86180>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-9h8ggzx7ljbh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-9h8ggzx7ljbh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc001b18840>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-ntrplhzxjq86/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc001b18840>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-ntrplhzxjq86/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0005ae660>: 
    failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-8zk685kvqsgq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8zk685kvqsgq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0005ae660>: 
    failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-8zk685kvqsgq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-8zk685kvqsgq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-1-7m29nrv4fdr6/cluster-nightly-5-1-scx85f should provision
Unexpected error:
    <*fmt.wrapError | 0xc0013fc280>: 
    failed to create HCP cluster cluster-nightly-5-1-scx85f: failed starting cluster creation "cluster-nightly-5-1-scx85f" in resourcegroup="rg-nightly-5-1-7m29nrv4fdr6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-7m29nrv4fdr6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-1-scx85f
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-1-7m29nrv4fdr6/cluster-nightly-5-1-scx85f should provision
Unexpected error:
    <*fmt.wrapError | 0xc0013fc280>: 
    failed to create HCP cluster cluster-nightly-5-1-scx85f: failed starting cluster creation "cluster-nightly-5-1-scx85f" in resourcegroup="rg-nightly-5-1-7m29nrv4fdr6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-7m29nrv4fdr6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-1-scx85f
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc001030240>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-42c2h57lg25f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-42c2h57lg25f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc001030240>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-42c2h57lg25f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-42c2h57lg25f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000682260>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-j585nqsnf67j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-j585nqsnf67j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000682260>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-j585nqsnf67j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-j585nqsnf67j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-c4n9pr with version 4.21.0-0.nightly-multi-2026-09-22-123437
Unexpected error:
    <*fmt.wrapError | 0xc000bb8780>: 
    failed to create HCP cluster np-version-upgrade-cluster-c4n9pr: failed starting cluster creation "np-version-upgrade-cluster-c4n9pr" in resourcegroup="rg-np-version-upgrade-c4n9pr-lvc25kgsdccb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-c4n9pr-lvc25kgsdccb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-c4n9pr
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-c4n9pr with version 4.21.0-0.nightly-multi-2026-09-22-123437
Unexpected error:
    <*fmt.wrapError | 0xc000bb8780>: 
    failed to create HCP cluster np-version-upgrade-cluster-c4n9pr: failed starting cluster creation "np-version-upgrade-cluster-c4n9pr" in resourcegroup="rg-np-version-upgrade-c4n9pr-lvc25kgsdccb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-c4n9pr-lvc25kgsdccb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-c4n9pr
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.22 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-22-r8dcl75vwsvm/cluster-nightly-4-22-rbw68x should provision
Unexpected error:
    <*fmt.wrapError | 0xc000714200>: 
    failed to create HCP cluster cluster-nightly-4-22-rbw68x: failed starting cluster creation "cluster-nightly-4-22-rbw68x" in resourcegroup="rg-nightly-4-22-r8dcl75vwsvm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-r8dcl75vwsvm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-22-rbw68x
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-22-r8dcl75vwsvm/cluster-nightly-4-22-rbw68x should provision
Unexpected error:
    <*fmt.wrapError | 0xc000714200>: 
    failed to create HCP cluster cluster-nightly-4-22-rbw68x: failed starting cluster creation "cluster-nightly-4-22-rbw68x" in resourcegroup="rg-nightly-4-22-r8dcl75vwsvm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-r8dcl75vwsvm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-22-rbw68x
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc00011b8a0>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-877p4644xp66": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-877p4644xp66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc00011b8a0>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-877p4644xp66": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-877p4644xp66/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000aec280>: 
    failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-ggdtclvwvh49": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-ggdtclvwvh49/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000aec280>: 
    failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-ggdtclvwvh49": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-ggdtclvwvh49/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to cordon, drain, and uncordon a node in an HCP cluster · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0001b3020>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-x64bd6kgw77d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-x64bd6kgw77d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0001b3020>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-x64bd6kgw77d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-x64bd6kgw77d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000e98240>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-nkc9f66h7mrb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-nkc9f66h7mrb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000e98240>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-nkc9f66h7mrb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-nkc9f66h7mrb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc00140c260>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-bmczqgpdvhmg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-bmczqgpdvhmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc00140c260>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-bmczqgpdvhmg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-bmczqgpdvhmg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc0010740e0>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-snrfdxp7lz6k": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-snrfdxp7lz6k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc0010740e0>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-snrfdxp7lz6k": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-snrfdxp7lz6k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000624460>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-72hqfj82n768": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-72hqfj82n768/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000624460>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-72hqfj82n768": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-72hqfj82n768/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc00114e160>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-bv97895wknf4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-bv97895wknf4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc00114e160>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-bv97895wknf4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-bv97895wknf4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000c82a40>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-b6pmc9sdknkf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-b6pmc9sdknkf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000c82a40>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-b6pmc9sdknkf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-b6pmc9sdknkf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc0013160e0>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-rx9wkcqgtnhz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-rx9wkcqgtnhz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc0013160e0>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-rx9wkcqgtnhz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-rx9wkcqgtnhz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.23 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-23-bzpzlqbzxqvl/cluster-nightly-4-23-p7gkv7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0009be140>: 
    failed to create HCP cluster cluster-nightly-4-23-p7gkv7: failed starting cluster creation "cluster-nightly-4-23-p7gkv7" in resourcegroup="rg-nightly-4-23-bzpzlqbzxqvl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-bzpzlqbzxqvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-23-p7gkv7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-23-bzpzlqbzxqvl/cluster-nightly-4-23-p7gkv7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0009be140>: 
    failed to create HCP cluster cluster-nightly-4-23-p7gkv7: failed starting cluster creation "cluster-nightly-4-23-p7gkv7" in resourcegroup="rg-nightly-4-23-bzpzlqbzxqvl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-bzpzlqbzxqvl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-23-p7gkv7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000886420>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-ps8hpf8pd6z9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-ps8hpf8pd6z9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000886420>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-ps8hpf8pd6z9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-ps8hpf8pd6z9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-frkjxj" at install version 5.0.0-0.nightly-multi-2026-09-21-210852
Unexpected error:
    <*fmt.wrapError | 0xc000bfe2e0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-frkjxj: failed starting cluster creation "cp-ystream-upgrade-5-1-frkjxj" in resourcegroup="rg-cp-ystream-upgrade-5-1-kcgn55zdv82f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-kcgn55zdv82f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-frkjxj
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-frkjxj" at install version 5.0.0-0.nightly-multi-2026-09-21-210852
Unexpected error:
    <*fmt.wrapError | 0xc000bfe2e0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-frkjxj: failed starting cluster creation "cp-ystream-upgrade-5-1-frkjxj" in resourcegroup="rg-cp-ystream-upgrade-5-1-kcgn55zdv82f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-kcgn55zdv82f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-frkjxj
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000744c40>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-2jnwvdzzh64d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-2jnwvdzzh64d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000744c40>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-2jnwvdzzh64d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-2jnwvdzzh64d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc001072300>: 
    failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-gtbrw7nrxjz2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-gtbrw7nrxjz2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc001072300>: 
    failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-gtbrw7nrxjz2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-gtbrw7nrxjz2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-kx2v6b" at install version 4.21.0-0.nightly-multi-2026-09-22-123437
Unexpected error:
    <*fmt.wrapError | 0xc000c83440>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-kx2v6b: failed starting cluster creation "cp-ystream-upgrade-4-22-kx2v6b" in resourcegroup="rg-cp-ystream-upgrade-4-22-sqqmsdwsqbrs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-sqqmsdwsqbrs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-kx2v6b
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-kx2v6b" at install version 4.21.0-0.nightly-multi-2026-09-22-123437
Unexpected error:
    <*fmt.wrapError | 0xc000c83440>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-kx2v6b: failed starting cluster creation "cp-ystream-upgrade-4-22-kx2v6b" in resourcegroup="rg-cp-ystream-upgrade-4-22-sqqmsdwsqbrs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-sqqmsdwsqbrs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-kx2v6b
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc001052180>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-khqv7gn8cz69": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-khqv7gn8cz69/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc001052180>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-khqv7gn8cz69": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-khqv7gn8cz69/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00102c3a0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-4qb52bqkpzjf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-4qb52bqkpzjf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00102c3a0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-4qb52bqkpzjf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-4qb52bqkpzjf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0006784e0>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-l2fgd6zqtf9h": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l2fgd6zqtf9h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0006784e0>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-l2fgd6zqtf9h": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-l2fgd6zqtf9h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

23:00:41 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.21 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-21-v8sz4sgjpztm/cluster-nightly-4-21-9lgzcb should provision
Unexpected error:
    <*fmt.wrapError | 0xc000cb22a0>: 
    failed to create HCP cluster cluster-nightly-4-21-9lgzcb: failed starting cluster creation "cluster-nightly-4-21-9lgzcb" in resourcegroup="rg-nightly-4-21-v8sz4sgjpztm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-v8sz4sgjpztm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-21-9lgzcb
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-21-v8sz4sgjpztm/cluster-nightly-4-21-9lgzcb should provision
Unexpected error:
    <*fmt.wrapError | 0xc000cb22a0>: 
    failed to create HCP cluster cluster-nightly-4-21-9lgzcb: failed starting cluster creation "cluster-nightly-4-21-9lgzcb" in resourcegroup="rg-nightly-4-21-v8sz4sgjpztm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-v8sz4sgjpztm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-21-9lgzcb
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

02:00:40

16m 10s

periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel

post-goodmerged PR

No additional run metadata captured

uksouth

e2e

failed

n/a

e2e (56)

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000f300e0>: 
    failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-rz6r289t49jf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-rz6r289t49jf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000f300e0>: 
    failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-rz6r289t49jf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-rz6r289t49jf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to cordon, drain, and uncordon a node in an HCP cluster · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b821e0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-474zgvs2k728": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-474zgvs2k728/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b821e0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-474zgvs2k728": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-474zgvs2k728/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-xbrtd7" with version 5.0.0-rc.2 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b30480>: 
    failed to create HCP cluster cluster-zstream-5-0-xbrtd7: failed starting cluster creation "cluster-zstream-5-0-xbrtd7" in resourcegroup="rg-zstream-upgrade-5-0-w6cvb247pkgq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-w6cvb247pkgq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xbrtd7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-xbrtd7" with version 5.0.0-rc.2 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b30480>: 
    failed to create HCP cluster cluster-zstream-5-0-xbrtd7: failed starting cluster creation "cluster-zstream-5-0-xbrtd7" in resourcegroup="rg-zstream-upgrade-5-0-w6cvb247pkgq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-w6cvb247pkgq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-xbrtd7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc000bcca20>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-zrxpwlk9jdzp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-zrxpwlk9jdzp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc000bcca20>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-zrxpwlk9jdzp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-zrxpwlk9jdzp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000f1e0e0>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-8fc9fdnjqp2w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-8fc9fdnjqp2w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000f1e0e0>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-8fc9fdnjqp2w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-8fc9fdnjqp2w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-h75t75dg5lvd/cluster-candidate-4-22-bc8pwv should provision
Unexpected error:
    <*fmt.wrapError | 0xc00155ce80>: 
    failed to create HCP cluster cluster-candidate-4-22-bc8pwv: failed starting cluster creation "cluster-candidate-4-22-bc8pwv" in resourcegroup="rg-candidate-4-22-h75t75dg5lvd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-h75t75dg5lvd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-22-bc8pwv
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-h75t75dg5lvd/cluster-candidate-4-22-bc8pwv should provision
Unexpected error:
    <*fmt.wrapError | 0xc00155ce80>: 
    failed to create HCP cluster cluster-candidate-4-22-bc8pwv: failed starting cluster creation "cluster-candidate-4-22-bc8pwv" in resourcegroup="rg-candidate-4-22-h75t75dg5lvd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-h75t75dg5lvd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-22-bc8pwv
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-5hgv94" with version 4.22.13 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000c6a200>: 
    failed to create HCP cluster cluster-zstream-4-22-5hgv94: failed starting cluster creation "cluster-zstream-4-22-5hgv94" in resourcegroup="rg-zstream-upgrade-4-22-rg8drsghclvc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-rg8drsghclvc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-5hgv94
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-5hgv94" with version 4.22.13 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000c6a200>: 
    failed to create HCP cluster cluster-zstream-4-22-5hgv94: failed starting cluster creation "cluster-zstream-4-22-5hgv94" in resourcegroup="rg-zstream-upgrade-4-22-rg8drsghclvc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-rg8drsghclvc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-5hgv94
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc0002a67a0>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-jdv6zjq9vb9z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-jdv6zjq9vb9z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc0002a67a0>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-jdv6zjq9vb9z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-jdv6zjq9vb9z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ba2aa0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-mgkv5vcnbk4t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-mgkv5vcnbk4t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ba2aa0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-mgkv5vcnbk4t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-mgkv5vcnbk4t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000b89040>: 
    failed to create HCP cluster ea-listnksgg8: failed starting cluster creation "ea-listnksgg8" in resourcegroup="ea-list-tx57g2bcffrp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-tx57g2bcffrp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listnksgg8
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000b89040>: 
    failed to create HCP cluster ea-listnksgg8: failed starting cluster creation "ea-listnksgg8" in resourcegroup="ea-list-tx57g2bcffrp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-tx57g2bcffrp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listnksgg8
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-zd2hv5 at version 4.22.14
Unexpected error:
    <*fmt.wrapError | 0xc001018dc0>: 
    failed to create HCP cluster np-skip-minor-zd2hv5: failed starting cluster creation "np-skip-minor-zd2hv5" in resourcegroup="rg-np-skip-minor-zd2hv5-xg6gmd9sxw8t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-zd2hv5-xg6gmd9sxw8t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-zd2hv5
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-zd2hv5 at version 4.22.14
Unexpected error:
    <*fmt.wrapError | 0xc001018dc0>: 
    failed to create HCP cluster np-skip-minor-zd2hv5: failed starting cluster creation "np-skip-minor-zd2hv5" in resourcegroup="rg-np-skip-minor-zd2hv5-xg6gmd9sxw8t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-zd2hv5-xg6gmd9sxw8t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-zd2hv5
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f0e200>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-2w6r47m2vzbz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-2w6r47m2vzbz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f0e200>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-2w6r47m2vzbz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-2w6r47m2vzbz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000640360>: 
    failed to create HCP cluster np-downgrade-k7lqv2: failed starting cluster creation "np-downgrade-k7lqv2" in resourcegroup="rg-np-downgrade-k7lqv2-wq4hwbkxc846": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-k7lqv2-wq4hwbkxc846/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-k7lqv2
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000640360>: 
    failed to create HCP cluster np-downgrade-k7lqv2: failed starting cluster creation "np-downgrade-k7lqv2" in resourcegroup="rg-np-downgrade-k7lqv2-wq4hwbkxc846": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-k7lqv2-wq4hwbkxc846/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-k7lqv2
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000707e00>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-pg4bl4582c6x": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-pg4bl4582c6x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000707e00>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-pg4bl4582c6x": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-pg4bl4582c6x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-6ltghn4pvj58/cluster-candidate-4-21-s5gc4g should provision
Unexpected error:
    <*fmt.wrapError | 0xc00072aea0>: 
    failed to create HCP cluster cluster-candidate-4-21-s5gc4g: failed starting cluster creation "cluster-candidate-4-21-s5gc4g" in resourcegroup="rg-candidate-4-21-6ltghn4pvj58": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-6ltghn4pvj58/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-21-s5gc4g
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-6ltghn4pvj58/cluster-candidate-4-21-s5gc4g should provision
Unexpected error:
    <*fmt.wrapError | 0xc00072aea0>: 
    failed to create HCP cluster cluster-candidate-4-21-s5gc4g: failed starting cluster creation "cluster-candidate-4-21-s5gc4g" in resourcegroup="rg-candidate-4-21-6ltghn4pvj58": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-6ltghn4pvj58/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-21-s5gc4g
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000eacda0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-4wwnsq5nbvwk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-4wwnsq5nbvwk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000eacda0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-4wwnsq5nbvwk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-4wwnsq5nbvwk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c242a0>: 
    failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-jh5p965r9h5p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jh5p965r9h5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c242a0>: 
    failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-jh5p965r9h5p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-jh5p965r9h5p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009f61e0>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-plvwdtqxskcm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-plvwdtqxskcm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009f61e0>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-plvwdtqxskcm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-plvwdtqxskcm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-hrgb26 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000f8a340>: 
    failed to create HCP cluster np-version-upgrade-cluster-hrgb26: failed starting cluster creation "np-version-upgrade-cluster-hrgb26" in resourcegroup="rg-np-version-upgrade-hrgb26-88r5lslmqpf7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-hrgb26-88r5lslmqpf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-hrgb26
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-hrgb26 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000f8a340>: 
    failed to create HCP cluster np-version-upgrade-cluster-hrgb26: failed starting cluster creation "np-version-upgrade-cluster-hrgb26" in resourcegroup="rg-np-version-upgrade-hrgb26-88r5lslmqpf7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-hrgb26-88r5lslmqpf7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-hrgb26
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f101e0>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-8m2tcg7xkh2f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-8m2tcg7xkh2f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f101e0>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-8m2tcg7xkh2f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-8m2tcg7xkh2f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000d920c0>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-dh726mjb6hfc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-dh726mjb6hfc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000d920c0>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-dh726mjb6hfc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-dh726mjb6hfc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc00083e4e0>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-mqdcg88p65p5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-mqdcg88p65p5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc00083e4e0>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-mqdcg88p65p5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-mqdcg88p65p5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000da21c0>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-wzckxjx8ln9d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-wzckxjx8ln9d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000da21c0>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-wzckxjx8ln9d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-wzckxjx8ln9d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-plbg5s" with version 4.20.37 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0014c45e0>: 
    failed to create HCP cluster cluster-zstream-4-20-plbg5s: failed starting cluster creation "cluster-zstream-4-20-plbg5s" in resourcegroup="rg-zstream-upgrade-4-20-5dctr2v29hxm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-5dctr2v29hxm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-plbg5s
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-plbg5s" with version 4.20.37 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0014c45e0>: 
    failed to create HCP cluster cluster-zstream-4-20-plbg5s: failed starting cluster creation "cluster-zstream-4-20-plbg5s" in resourcegroup="rg-zstream-upgrade-4-20-5dctr2v29hxm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-5dctr2v29hxm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-plbg5s
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000726720>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-6mvbgcknfnql": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6mvbgcknfnql/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000726720>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-6mvbgcknfnql": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-6mvbgcknfnql/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.20 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc0010ca2a0>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-wjdnbdcz7v5v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-wjdnbdcz7v5v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc0010ca2a0>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-wjdnbdcz7v5v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-wjdnbdcz7v5v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to delete an HCP cluster whose managed identities were deleted first · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f341e0>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-k4jvltkd7tqn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-k4jvltkd7tqn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f341e0>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-k4jvltkd7tqn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-k4jvltkd7tqn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc0006f8420>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-chbpbwgpb8pr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-chbpbwgpb8pr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc0006f8420>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-chbpbwgpb8pr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-chbpbwgpb8pr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-xvzwmw with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000f141a0>: 
    failed to create HCP cluster np-version-upgrade-cluster-xvzwmw: failed starting cluster creation "np-version-upgrade-cluster-xvzwmw" in resourcegroup="rg-np-version-upgrade-xvzwmw-bxjhmmlcjtxn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-xvzwmw-bxjhmmlcjtxn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-xvzwmw
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-xvzwmw with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000f141a0>: 
    failed to create HCP cluster np-version-upgrade-cluster-xvzwmw: failed starting cluster creation "np-version-upgrade-cluster-xvzwmw" in resourcegroup="rg-np-version-upgrade-xvzwmw-bxjhmmlcjtxn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-xvzwmw-bxjhmmlcjtxn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-xvzwmw
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000b48360>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-84h94kdhhq2z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-84h94kdhhq2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000b48360>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-84h94kdhhq2z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-84h94kdhhq2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001000160>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-lbjqxnvvbh9l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-lbjqxnvvbh9l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001000160>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-lbjqxnvvbh9l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-lbjqxnvvbh9l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000d26300>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-kxlbk5nhx7dg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-kxlbk5nhx7dg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000d26300>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-kxlbk5nhx7dg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-kxlbk5nhx7dg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000aaa160>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-ww5k9bfrs47z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-ww5k9bfrs47z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000aaa160>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-ww5k9bfrs47z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-ww5k9bfrs47z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc002880260>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-98wggrdxqbzz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-98wggrdxqbzz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc002880260>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-98wggrdxqbzz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-98wggrdxqbzz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc00103c2a0>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-lhpbhxvtfvqm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-lhpbhxvtfvqm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc00103c2a0>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-lhpbhxvtfvqm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-lhpbhxvtfvqm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-j8stln" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc001092140>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-j8stln: failed starting cluster creation "cp-ystream-upgrade-4-21-j8stln" in resourcegroup="rg-cp-ystream-upgrade-4-21-k6fzpk2b68m4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-k6fzpk2b68m4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-j8stln
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-j8stln" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc001092140>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-j8stln: failed starting cluster creation "cp-ystream-upgrade-4-21-j8stln" in resourcegroup="rg-cp-ystream-upgrade-4-21-k6fzpk2b68m4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-k6fzpk2b68m4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-j8stln
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc00118a260>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-q464fsngnb2z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-q464fsngnb2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc00118a260>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-q464fsngnb2z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-q464fsngnb2z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-mckpkv
Unexpected error:
    <*fmt.wrapError | 0xc000860720>: 
    failed to create HCP cluster np-noedge-mckpkv: failed starting cluster creation "np-noedge-mckpkv" in resourcegroup="rg-np-noedge-mckpkv-chk6hnblbkth": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-mckpkv-chk6hnblbkth/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-mckpkv
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-mckpkv
Unexpected error:
    <*fmt.wrapError | 0xc000860720>: 
    failed to create HCP cluster np-noedge-mckpkv: failed starting cluster creation "np-noedge-mckpkv" in resourcegroup="rg-np-noedge-mckpkv-chk6hnblbkth": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-mckpkv-chk6hnblbkth/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-mckpkv
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc0010303c0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-6t49grdzfqmk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc0010303c0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-6t49grdzfqmk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc001012360>: 
    failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-d5hfbw5gmtt5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-d5hfbw5gmtt5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc001012360>: 
    failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-d5hfbw5gmtt5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-d5hfbw5gmtt5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc00062c5e0>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-lczl6v6nrz6w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-lczl6v6nrz6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc00062c5e0>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-lczl6v6nrz6w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-lczl6v6nrz6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc00092c680>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-czpph9wzgls4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-czpph9wzgls4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc00092c680>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-czpph9wzgls4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-czpph9wzgls4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to list available HCP OpenShift versions and validate response content · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc001757c20>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc001757c20>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000b1af80>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-v2bhlsm92ksh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-v2bhlsm92ksh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000b1af80>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-v2bhlsm92ksh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-v2bhlsm92ksh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-cxl549" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc0011c8240>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-cxl549: failed starting cluster creation "cp-ystream-upgrade-5-0-cxl549" in resourcegroup="rg-cp-ystream-upgrade-5-0-klrb5nkdm6fv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-klrb5nkdm6fv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-cxl549
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-cxl549" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc0011c8240>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-cxl549: failed starting cluster creation "cp-ystream-upgrade-5-0-cxl549" in resourcegroup="rg-cp-ystream-upgrade-5-0-klrb5nkdm6fv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-klrb5nkdm6fv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-cxl549
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-c2r948" with version 4.21.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000ed43c0>: 
    failed to create HCP cluster cluster-zstream-4-21-c2r948: failed starting cluster creation "cluster-zstream-4-21-c2r948" in resourcegroup="rg-zstream-upgrade-4-21-swxb5fxkxx4z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-swxb5fxkxx4z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-c2r948
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-c2r948" with version 4.21.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000ed43c0>: 
    failed to create HCP cluster cluster-zstream-4-21-c2r948: failed starting cluster creation "cluster-zstream-4-21-c2r948" in resourcegroup="rg-zstream-upgrade-4-21-swxb5fxkxx4z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-swxb5fxkxx4z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-c2r948
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000fde240>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-qj86j7vf296r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-qj86j7vf296r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000fde240>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-qj86j7vf296r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-qj86j7vf296r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

failed to create HCP cluster <cluster>: failed starting cluster creation "<cluster>" in resourcegroup="<resource-group>": Put "<url>": stream error: stream ID 37; CANCEL; received from peer

02:00:40 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-6bkkt2 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc0013940e0>: 
    failed to create HCP cluster np-version-upgrade-cluster-6bkkt2: failed starting cluster creation "np-version-upgrade-cluster-6bkkt2" in resourcegroup="rg-np-version-upgrade-6bkkt2-m96mr2dxx5b8": Put "https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-6bkkt2-m96mr2dxx5b8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-6bkkt2?api-version=2024-06-10-preview": stream error: stream ID 37; CANCEL; received from peer
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-6bkkt2 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc0013940e0>: 
    failed to create HCP cluster np-version-upgrade-cluster-6bkkt2: failed starting cluster creation "np-version-upgrade-cluster-6bkkt2" in resourcegroup="rg-np-version-upgrade-6bkkt2-m96mr2dxx5b8": Put "https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-6bkkt2-m96mr2dxx5b8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-6bkkt2?api-version=2024-06-10-preview": stream error: stream ID 37; CANCEL; received from peer
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc00113a320>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-kbl8bmnn4b4w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-kbl8bmnn4b4w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc00113a320>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-kbl8bmnn4b4w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-kbl8bmnn4b4w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-g28hnvmc7v6c/cluster-candidate-4-20-762dl6 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ca4160>: 
    failed to create HCP cluster cluster-candidate-4-20-762dl6: failed starting cluster creation "cluster-candidate-4-20-762dl6" in resourcegroup="rg-candidate-4-20-g28hnvmc7v6c": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-g28hnvmc7v6c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-20-762dl6
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-g28hnvmc7v6c/cluster-candidate-4-20-762dl6 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ca4160>: 
    failed to create HCP cluster cluster-candidate-4-20-762dl6: failed starting cluster creation "cluster-candidate-4-20-762dl6" in resourcegroup="rg-candidate-4-20-g28hnvmc7v6c": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-g28hnvmc7v6c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-20-762dl6
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000da2400>: 
    failed to create HCP cluster np-dg-minor-qh8fbr: failed starting cluster creation "np-dg-minor-qh8fbr" in resourcegroup="rg-np-dg-minor-qh8fbr-m5zhfcgf52j7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-qh8fbr-m5zhfcgf52j7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-qh8fbr
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000da2400>: 
    failed to create HCP cluster np-dg-minor-qh8fbr: failed starting cluster creation "np-dg-minor-qh8fbr" in resourcegroup="rg-np-dg-minor-qh8fbr-m5zhfcgf52j7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-qh8fbr-m5zhfcgf52j7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-qh8fbr
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-h79k8k" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc0014d33a0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-h79k8k: failed starting cluster creation "cp-ystream-upgrade-4-22-h79k8k" in resourcegroup="rg-cp-ystream-upgrade-4-22-w48jfg4cknrl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-w48jfg4cknrl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-h79k8k
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-h79k8k" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc0014d33a0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-h79k8k: failed starting cluster creation "cp-ystream-upgrade-4-22-h79k8k" in resourcegroup="rg-cp-ystream-upgrade-4-22-w48jfg4cknrl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-w48jfg4cknrl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-h79k8k
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-pxrfzhr4lgdl/cluster-candidate-5-0-gsk4z7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00097ccc0>: 
    failed to create HCP cluster cluster-candidate-5-0-gsk4z7: failed starting cluster creation "cluster-candidate-5-0-gsk4z7" in resourcegroup="rg-candidate-5-0-pxrfzhr4lgdl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-pxrfzhr4lgdl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-gsk4z7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-pxrfzhr4lgdl/cluster-candidate-5-0-gsk4z7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00097ccc0>: 
    failed to create HCP cluster cluster-candidate-5-0-gsk4z7: failed starting cluster creation "cluster-candidate-5-0-gsk4z7" in resourcegroup="rg-candidate-5-0-pxrfzhr4lgdl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-pxrfzhr4lgdl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-gsk4z7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:105]: failed to begin creating HCP cluster "admin-cred-lifecycle-q52tkz"
Unexpected error:
    <*fmt.wrapError | 0xc001098400>: 
    failed starting cluster creation "admin-cred-lifecycle-q52tkz" in resourcegroup="admin-credential-lifecycle-test-ccbpf2vrpk7m": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-ccbpf2vrpk7m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-q52tkz
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:105]: failed to begin creating HCP cluster "admin-cred-lifecycle-q52tkz"
Unexpected error:
    <*fmt.wrapError | 0xc001098400>: 
    failed starting cluster creation "admin-cred-lifecycle-q52tkz" in resourcegroup="admin-credential-lifecycle-test-ccbpf2vrpk7m": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-ccbpf2vrpk7m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-q52tkz
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000a6e540>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-lq7mxkdms4nx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-lq7mxkdms4nx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000a6e540>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-lq7mxkdms4nx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-lq7mxkdms4nx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift

02:00:40 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite stage/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001956280>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-xfb8qbsjtxct": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-xfb8qbsjtxct/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001956280>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-xfb8qbsjtxct": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-xfb8qbsjtxct/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred