alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T23:47:51Z
  Ended:       2026-09-19T00:19:53Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j8441088-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j8441088-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000b4b300>: 
    failed to create HCP cluster ea-listtxmfvt: failed waiting for cluster="ea-listtxmfvt" in resourcegroup="ea-list-b22vvl6sfzgh" to finish creating: GET https://rp.j8565632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/80b95276-553d-4fcb-9838-22d1a7553049
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/80b95276-553d-4fcb-9838-22d1a7553049",
      "name": "80b95276-553d-4fcb-9838-22d1a7553049",
      "status": "Failed",
      "startTime": "2026-09-18T23:30:18.658850049Z",
      "endTime": "2026-09-18T23:59:19.341659624Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000b4b300>: 
    failed to create HCP cluster ea-listtxmfvt: failed waiting for cluster="ea-listtxmfvt" in resourcegroup="ea-list-b22vvl6sfzgh" to finish creating: GET https://rp.j8565632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/80b95276-553d-4fcb-9838-22d1a7553049
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/80b95276-553d-4fcb-9838-22d1a7553049",
      "name": "80b95276-553d-4fcb-9838-22d1a7553049",
      "status": "Failed",
      "startTime": "2026-09-18T23:30:18.658850049Z",
      "endTime": "2026-09-18T23:59:19.341659624Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T00:04:52Z
  Ended:       2026-09-19T00:10:51Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8565632-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="governance-policy-framework", endpoint="http", environment="ci01", instance="10.128.64.153:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8565632/providers/microsoft.monitor/accounts/services-j8565632", namespace="klusterlet-2stnn2qj5n4300vnra0q4o64lcoi8994", pod="arohcp-monitor-kube-state-metrics-6486647766-v6bbh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2stnn2qj5n4300vnra0q4o64lcoi8994/governance-policy-framework has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-19T00:04:52Z
  Ended:       2026-09-19T00:10:50Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8565632-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="config-policy-controller", endpoint="http", environment="ci01", instance="10.128.64.153:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8565632/providers/microsoft.monitor/accounts/services-j8565632", namespace="klusterlet-2stnn2qj5n4300vnra0q4o64lcoi8994", pod="arohcp-monitor-kube-state-metrics-6486647766-v6bbh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2stnn2qj5n4300vnra0q4o64lcoi8994/config-policy-controller has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T23:50:30Z
  Ended:       2026-09-19T00:17:29Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8565632-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.153:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8565632/providers/microsoft.monitor/accounts/hcps-j8565632", namespace="ocm-arohcpci01-2stnn2qj5n4300vnra0q4o64lcoi8994-ea-listtxmfvt", pod="arohcp-monitor-kube-state-metrics-6486647766-v6bbh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2stnn2qj5n4300vnra0q4o64lcoi8994-ea-listtxmfvt/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-18T23:50:30Z
  Ended:       2026-09-19T00:18:33Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8565632-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.153:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8565632/providers/microsoft.monitor/accounts/hcps-j8565632", namespace="ocm-arohcpci01-2stnn2qj5n4300vnra0q4o64lcoi8994-ea-listtxmfvt", pod="arohcp-monitor-kube-state-metrics-6486647766-v6bbh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2stnn2qj5n4300vnra0q4o64lcoi8994-ea-listtxmfvt/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T00:04:30Z
  Ended:       2026-09-19T00:23:35Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8565632-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.64.153:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8565632/providers/microsoft.monitor/accounts/hcps-j8565632", namespace="ocm-arohcpci01-2stnn2qj5n4300vnra0q4o64lcoi8994-ea-listtxmfvt", pod="arohcp-monitor-kube-state-metrics-6486647766-v6bbh", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2stnn2qj5n4300vnra0q4o64lcoi8994-ea-listtxmfvt/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T23:52:29Z
  Ended:       2026-09-19T00:21:32Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8565632-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.153:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8565632/providers/microsoft.monitor/accounts/hcps-j8565632", namespace="ocm-arohcpci01-2stnn2qj5n4300vnra0q4o64lcoi8994-ea-listtxmfvt", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e1a1529f-88f5-4f8c-8503-a2513cf48144"
  Description: Pod ocm-arohcpci01-2stnn2qj5n4300vnra0q4o64lcoi8994-ea-listtxmfvt/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-18T23:53:28Z
  Ended:       2026-09-19T00:17:27Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8565632-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.153:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8565632/providers/microsoft.monitor/accounts/hcps-j8565632", namespace="ocm-arohcpci01-2stnn2qj5n4300vnra0q4o64lcoi8994-ea-listtxmfvt", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e1a1529f-88f5-4f8c-8503-a2513cf48144"
  Description: Pod ocm-arohcpci01-2stnn2qj5n4300vnra0q4o64lcoi8994-ea-listtxmfvt/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").

time=2026-09-18T21:50:35.372Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring description="Step monitoring\n  Kind: ARMStack\n  Template: templates/monitoring.bicep\n  Parameters: configurations/monitoring.tmpl.bicepparam"
time=2026-09-18T21:51:25.579Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring err="failed to run ARM step: failed to wait for deployment stack at resource group scope: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/westus2/deploymentStackOperationStatus/04392d3b-593d-406d-ae87-240d0d6c1c70\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentStackDeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/westus2/deploymentStackOperationStatus/04392d3b-593d-406d-ae87-240d0d6c1c70\",\n  \"name\": \"04392d3b-593d-406d-ae87-240d0d6c1c70\",\n  \"status\": \"failed\",\n  \"error\": {\n    \"code\": \"DeploymentStackDeploymentFailed\",\n    \"message\": \"One or more resources could not be deployed. Correlation id: 'aad2e336-6c43-473c-997c-4a60a482583a'.\",\n    \"details\": [\n      {\n        \"code\": \"DeploymentFailed\",\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4937088/providers/Microsoft.Resources/deployments/y5flko475afn22dntfyewzfj7gfkbiwdmj7ezjquufwse-26091821etad3\",\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n        \"details\": [\n          {\n            \"code\": \"BadRequest\",\n            \"message\": \"Property name: Rules[0].Expression, Attempted value: null, Error: 1:237: parse error: unknown escape sequence U+002E '.' Activity ID: b8850252-9aac-448f-9581-e29b2b8a10a8.\"\n          }\n        ]\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

time=2026-09-18T21:55:47.666Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy description="Step deploy\n Kind: Helm\n"
time=2026-09-18T21:55:48.312Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy values="map[armPermissionsManagerIdentity:map[certName:armHelperPoolCert-6 clientId:9c78a811-9c85-4eb0-9c84-d0843866c888 tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] azure:map[cloudEnvironmentName:AzurePublicCloud msiRpDataPlaneAudienceResource:https://dummy.org ocpAcrHostname:arohcpocpdev.azurecr.io ocpAcrResourceId:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/global/providers/Microsoft.ContainerRegistry/registries/arohcpocpdev oidcIssuerBaseUrl:https://arohcpci01oidcj1086464.z1.web.core.windows.net/ oidcIssuerBlobServiceUrl:https://arohcpci01oidcj1086464.blob.core.windows.net/ tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX tlsCertificatesConfig:map[certificatesGenerationSource:AzureKeyVault issuer:Self]] backupCadenceProfile:testing backupScheduleState:Enabled clusterScopedIdentitiesConfig:map[roleSetName:dev] clustersService:map[namespace:clusters-service serviceAccount:clusters-service] configMap:map[backendMiClientId:780fc8ab-b71d-4dc8-a753-5e608208a59f databaseName:arohcpci01-rp-j1086464 databaseUrl:https://arohcpci01-rp-j1086464.documents.azure.com:443/ kubeApplierCosmosContainerName:Manifests-MC-1 location:westus3] deployment:map[imageName:arohcpsvcdev.azurecr.io/ci-op-vh2hkbkg/pipeline@sha256:ec4f5311c601606f0bdcb4316ea9dff923b912c8e13a93addd5ac2c23dd9e86c replicas:2 resources:map[limits:map[memory:unlimited] requests:map[cpu:475m memory:1Gi]] zoneCount:3] exitOnPanic:true fpa:map[certName:firstPartyCert2 clientId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] logVerbosity:4 maestro:map[sourceEnvironmentIdentifier:arohcpci01] miMock:map[certName:msiMockPoolCert-12 clientId:50d052a6-7aec-48b9-b43e-b8692a8f0ead clientIdSecretName: principalId:a76148d4-cb11-4b93-9363-54ba8239b0b1 principalIdSecretName: tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] pullBinding:map[registry:arohcpsvcdev.azurecr.io scope:repository:ci-op-vh2hkbkg/pipeline:pull workloadIdentityClientId:09708096-ca85-4992-8eb5-667a9b9d3fce workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceAccount:map[name:backend workloadIdentityClientId:780fc8ab-b71d-4dc8-a753-5e608208a59f workloadIdentityTenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] serviceKeyVault:map[name:aro-hcp-dev-svc-kv] tracing:map[address:http://ingest.observability:4318 exporter:otlp]]"
time=2026-09-18T21:55:48.312Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-09-18T21:55:48.698Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy namespace=aro-hcp
time=2026-09-18T21:55:48.964Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy dryRun=false
time=2026-09-18T21:55:48.964Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-09-18T21:55:49.102Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy
time=2026-09-18T22:00:55.361Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="resource Deployment/aro-hcp/aro-hcp-backend not ready. status: InProgress, message: Available: 1/2\ncontext deadline exceeded"
time=2026-09-18T22:00:55.361Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.RP.Backend resourceGroup=service step=deploy err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/aro-hcp/aro-hcp-backend not ready. status: InProgress, message: Available: 1/2\ncontext deadline exceeded"

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.009s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.009s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: ...
to be true

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:383]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc001bed8b0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:383]: Timed out after 600.001s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc001bed8b0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T22:08:21Z
  Ended:       2026-09-18T22:18:16Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j4790528-svc", component="frontend", method="delete", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: More than 5% of frontend requests for delete /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j4790528-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T21:03:41Z
  Ended:       2026-09-18T21:09:42Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6750464-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6750464-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc000dd1600>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-bjbkvcsbld8c" to finish creating: GET https://rp.j0272768.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/641e2016-08be-40cf-9f8e-0e897bdccd1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/641e2016-08be-40cf-9f8e-0e897bdccd1d",
      "name": "641e2016-08be-40cf-9f8e-0e897bdccd1d",
      "status": "Failed",
      "startTime": "2026-09-18T20:46:36.342825036Z",
      "endTime": "2026-09-18T21:05:43.553710506Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:222]: failed to create nodepool "np-1" for authorized CIDRs cluster
Unexpected error:
    <*fmt.wrapError | 0xc000dd1600>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-bjbkvcsbld8c" to finish creating: GET https://rp.j0272768.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/641e2016-08be-40cf-9f8e-0e897bdccd1d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/641e2016-08be-40cf-9f8e-0e897bdccd1d",
      "name": "641e2016-08be-40cf-9f8e-0e897bdccd1d",
      "status": "Failed",
      "startTime": "2026-09-18T20:46:36.342825036Z",
      "endTime": "2026-09-18T21:05:43.553710506Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

time=2026-09-18T20:01:01.932Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kube-events stamp=1 description="Step kube-events\n Kind: Helm\n"
time=2026-09-18T20:01:02.379Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kube-events stamp=1 values="map[enabled:true image:map[digest:sha256:5177425b194b5d289664c62bedbdc95a42605bbfcde48fc7ddbb31bee0194b90 registry:kubernetesshared.azurecr.io repository:shared/kube-events] resources:map[limits:map[memory:unlimited] requests:map[cpu:200m memory:512Mi]]]"
time=2026-09-18T20:01:02.379Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kube-events stamp=1
time=2026-09-18T20:01:02.574Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kube-events stamp=1 namespace=monitoring
time=2026-09-18T20:01:02.761Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kube-events stamp=1 dryRun=false
time=2026-09-18T20:01:02.761Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kube-events stamp=1
time=2026-09-18T20:01:02.827Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kube-events stamp=1
time=2026-09-18T20:06:05.062Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kube-events stamp=1 err="resource Deployment/monitoring/kube-events not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"
time=2026-09-18T20:06:05.062Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kube-events stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource Deployment/monitoring/kube-events not ready. status: InProgress, message: Available: 0/1\ncontext deadline exceeded"

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:188]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-v458r6"
Unexpected error:
    <*fmt.wrapError | 0xc001029700>: 
    failed to create route: the server is currently unable to handle the request
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:188]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-v458r6"
Unexpected error:
    <*fmt.wrapError | 0xc001029700>: 
    failed to create route: the server is currently unable to handle the request
    ...
occurred

alert fired 13 time(s) (1 unknown, 12 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T20:11:59Z
  Ended:       2026-09-18T20:26:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6109056-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2stkk28pfi15i802bl7aop2360tp9eph-a9i6s2l5j1h0r1s", pod="router-8444b6f56d-w6qnw", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2stkk28pfi15i802bl7aop2360tp9eph-a9i6s2l5j1h0r1s/router-8444b6f56d-w6qnw has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc00255de60>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-8ld2f474grdt" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-8ld2f474grdt, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:186]: failed to update node pool np-1 with new taints and scale up
Unexpected error:
    <*fmt.wrapErrors | 0xc00255de60>: 
    failed waiting for nodepool="np-1" in cluster="cluster-np-labels-taints" resourcegroup="rg-np-labels-taints-8ld2f474grdt" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-1 in cluster cluster-np-labels-taints in resource group rg-np-labels-taints-8ld2f474grdt, error: context deadline exceeded
    ...
occurred

time=2026-09-18T18:49:43.745Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra-output description="Step infra-output\n  Kind: ARM\n  Template: templates/svc-infra-lookup.bicep\n  Parameters: configurations/svc-infra-lookup.tmpl.bicepparam"
time=2026-09-18T18:49:44.412Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra-output
time=2026-09-18T18:49:45.666Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra-output deployment=299b33c33a8d1927824771d1411d9cc2911c3c4239fa63e4af7c6c883950aabd portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j6366720-svc%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F299b33c33a8d1927824771d1411d9cc2911c3c4239fa63e4af7c6c883950aabd
time=2026-09-18T18:50:16.326Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=infra-output err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6366720-svc/providers/Microsoft.Resources/deployments/299b33c33a8d1927824771d1411d9cc2911c3c4239fa63e4af7c6c883950aabd/operationStatuses/08584118495004368821\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"NotFound\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"DeploymentNotFound\\\",\\r\\n    \\\"message\\\": \\\"Deployment 'svc-kv-xehipxbjxawki' could not be found.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000ce8120>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-4k8qcm9nwvt9" to finish creating: GET https://rp.j9556480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ad50471-887d-48c6-8d40-87da15772f0f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ad50471-887d-48c6-8d40-87da15772f0f",
      "name": "8ad50471-887d-48c6-8d40-87da15772f0f",
      "status": "Failed",
      "startTime": "2026-09-18T17:24:52.179447145Z",
      "endTime": "2026-09-18T17:53:56.374800037Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m13s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000ce8120>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-4k8qcm9nwvt9" to finish creating: GET https://rp.j9556480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ad50471-887d-48c6-8d40-87da15772f0f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ad50471-887d-48c6-8d40-87da15772f0f",
      "name": "8ad50471-887d-48c6-8d40-87da15772f0f",
      "status": "Failed",
      "startTime": "2026-09-18T17:24:52.179447145Z",
      "endTime": "2026-09-18T17:53:56.374800037Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m13s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T17:37:21Z
  Ended:       2026-09-18T18:05:16Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9556480-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2stibp3tbntp3l84dd3nd1p1jo0f3edi-private-ingress", pod="router-8495fb9c6b-n57p2", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2stibp3tbntp3l84dd3nd1p1jo0f3edi-private-ingress/router-8495fb9c6b-n57p2 has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T17:56:41Z
  Ended:       2026-09-18T18:02:41Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9556480-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9556480-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T18:04:41Z
  Ended:       2026-09-18T18:24:40Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j9556480-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j9556480-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T17:42:43Z
  Ended:       2026-09-18T17:48:43Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3570304-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3570304-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T17:59:03Z
  Ended:       2026-09-18T18:05:04Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j2544640-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j2544640-svc).

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012e61b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-ghmpzk" in resourcegroup="rg-zstream-upgrade-4-20-nzh4xtf2qkhs" to finish deleting: GET https://rp.j1089664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/58c027c6-4639-48fa-ace0-f454c7358b62
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/58c027c6-4639-48fa-ace0-f454c7358b62",
      "name": "58c027c6-4639-48fa-ace0-f454c7358b62",
      "status": "Failed",
      "startTime": "2026-09-18T17:43:42.257649954Z",
      "endTime": "2026-09-18T18:07:51.061074345Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sti7ae2rm414v72bb1h0rpr5prt24in still exists (deletion dispatched at 2026-09-18T17:43:42Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0012e61b0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="cluster-zstream-4-20-ghmpzk" in resourcegroup="rg-zstream-upgrade-4-20-nzh4xtf2qkhs" to finish deleting: GET https://rp.j1089664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/58c027c6-4639-48fa-ace0-f454c7358b62
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/58c027c6-4639-48fa-ace0-f454c7358b62",
      "name": "58c027c6-4639-48fa-ace0-f454c7358b62",
      "status": "Failed",
      "startTime": "2026-09-18T17:43:42.257649954Z",
      "endTime": "2026-09-18T18:07:51.061074345Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sti7ae2rm414v72bb1h0rpr5prt24in still exists (deletion dispatched at 2026-09-18T17:43:42Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0014723f0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listg6frkr" in resourcegroup="ea-list-r7jhwkxx4sdv" to finish deleting: GET https://rp.j1089664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9bf9f7c9-1f78-4d84-bb7d-1c79558f8921
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9bf9f7c9-1f78-4d84-bb7d-1c79558f8921",
      "name": "9bf9f7c9-1f78-4d84-bb7d-1c79558f8921",
      "status": "Failed",
      "startTime": "2026-09-18T17:26:18.493088022Z",
      "endTime": "2026-09-18T17:50:21.112379819Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sti7eotueh3r9mn8c08fdvae8djl7o8 still exists (deletion dispatched at 2026-09-18T17:26:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0014723f0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="ea-listg6frkr" in resourcegroup="ea-list-r7jhwkxx4sdv" to finish deleting: GET https://rp.j1089664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9bf9f7c9-1f78-4d84-bb7d-1c79558f8921
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/9bf9f7c9-1f78-4d84-bb7d-1c79558f8921",
      "name": "9bf9f7c9-1f78-4d84-bb7d-1c79558f8921",
      "status": "Failed",
      "startTime": "2026-09-18T17:26:18.493088022Z",
      "endTime": "2026-09-18T17:50:21.112379819Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sti7eotueh3r9mn8c08fdvae8djl7o8 still exists (deletion dispatched at 2026-09-18T17:26:19Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00156c060>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-nzl72g5md8zn" to finish deleting: GET https://rp.j1089664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a6fd0546-7edc-4a7f-8a5d-bb281048b13a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a6fd0546-7edc-4a7f-8a5d-bb281048b13a",
      "name": "a6fd0546-7edc-4a7f-8a5d-bb281048b13a",
      "status": "Failed",
      "startTime": "2026-09-18T17:31:17.188166352Z",
      "endTime": "2026-09-18T17:55:21.133690065Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sti7a39k8no3d3k6o47iqafudkp0ufb still exists (deletion dispatched at 2026-09-18T17:31:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc00156c060>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="sre-hcp-cluster" in resourcegroup="admin-api-breakglass-nzl72g5md8zn" to finish deleting: GET https://rp.j1089664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a6fd0546-7edc-4a7f-8a5d-bb281048b13a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a6fd0546-7edc-4a7f-8a5d-bb281048b13a",
      "name": "a6fd0546-7edc-4a7f-8a5d-bb281048b13a",
      "status": "Failed",
      "startTime": "2026-09-18T17:31:17.188166352Z",
      "endTime": "2026-09-18T17:55:21.133690065Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sti7a39k8no3d3k6o47iqafudkp0ufb still exists (deletion dispatched at 2026-09-18T17:31:17Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters; [resourceTeardown] tearing down namespaces (1)"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0017e8150>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-5mwmp59kkrll" to finish deleting: GET https://rp.j1089664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0000780c-679b-479e-b61c-e08211991339
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0000780c-679b-479e-b61c-e08211991339",
      "name": "0000780c-679b-479e-b61c-e08211991339",
      "status": "Failed",
      "startTime": "2026-09-18T17:27:09.508692971Z",
      "endTime": "2026-09-18T17:51:11.117212687Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sti77pkq7agiit4e9p66ic3479hnrq5 still exists (deletion dispatched at 2026-09-18T17:27:10Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0017e8150>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-5mwmp59kkrll" to finish deleting: GET https://rp.j1089664.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0000780c-679b-479e-b61c-e08211991339
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/0000780c-679b-479e-b61c-e08211991339",
      "name": "0000780c-679b-479e-b61c-e08211991339",
      "status": "Failed",
      "startTime": "2026-09-18T17:27:09.508692971Z",
      "endTime": "2026-09-18T17:51:11.117212687Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2sti77pkq7agiit4e9p66ic3479hnrq5 still exists (deletion dispatched at 2026-09-18T17:27:10Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 6 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-18T18:02:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1089664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="router", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1089664/providers/microsoft.monitor/accounts/hcps-j1089664", namespace="ocm-arohcpci01-2sti77pkq7agiit4e9p66ic3479hnrq5-a7o9q7y9a3g4n2w", pod="arohcp-monitor-kube-state-metrics-6486647766-6vsrg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sti77pkq7agiit4e9p66ic3479hnrq5-a7o9q7y9a3g4n2w/router has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-18T18:03:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1089664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="router", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1089664/providers/microsoft.monitor/accounts/hcps-j1089664", namespace="ocm-arohcpci01-2sti77pkq7agiit4e9p66ic3479hnrq5-a7o9q7y9a3g4n2w", pod="arohcp-monitor-kube-state-metrics-6486647766-6vsrg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sti77pkq7agiit4e9p66ic3479hnrq5-a7o9q7y9a3g4n2w/router has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-18T18:11:31Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1089664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1089664/providers/microsoft.monitor/accounts/hcps-j1089664", namespace="ocm-arohcpci01-2sti7a39k8no3d3k6o47iqafudkp0ufb-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-6486647766-6vsrg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sti7a39k8no3d3k6o47iqafudkp0ufb-sre-hcp-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-18T18:11:31Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1089664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1089664/providers/microsoft.monitor/accounts/hcps-j1089664", namespace="ocm-arohcpci01-2sti7a39k8no3d3k6o47iqafudkp0ufb-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-6486647766-6vsrg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sti7a39k8no3d3k6o47iqafudkp0ufb-sre-hcp-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Fired (not resolved)
  Started:     2026-09-18T18:12:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1089664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="cluster-api", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1089664/providers/microsoft.monitor/accounts/hcps-j1089664", namespace="ocm-arohcpci01-2sti7a39k8no3d3k6o47iqafudkp0ufb-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-6486647766-6vsrg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sti7a39k8no3d3k6o47iqafudkp0ufb-sre-hcp-cluster/cluster-api has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Fired (not resolved)
  Started:     2026-09-18T18:12:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1089664-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1089664/providers/microsoft.monitor/accounts/hcps-j1089664", namespace="ocm-arohcpci01-2sti7a39k8no3d3k6o47iqafudkp0ufb-sre-hcp-cluster", pod="arohcp-monitor-kube-state-metrics-6486647766-6vsrg", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2sti7a39k8no3d3k6o47iqafudkp0ufb-sre-hcp-cluster/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 3 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-18T18:04:42Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j1089664-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", lease="work-manager", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1089664/providers/microsoft.monitor/accounts/services-j1089664", namespace="klusterlet-2sti77pkq7agiit4e9p66ic3479hnrq5", pod="arohcp-monitor-kube-state-metrics-6486647766-6vsrg", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease work-manager in namespace klusterlet-2sti77pkq7agiit4e9p66ic3479hnrq5 on cluster ci01-j1089664-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-18T18:05:44Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j1089664-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", lease="governance-policy-framework", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1089664/providers/microsoft.monitor/accounts/services-j1089664", namespace="klusterlet-2sti77pkq7agiit4e9p66ic3479hnrq5", pod="arohcp-monitor-kube-state-metrics-6486647766-6vsrg", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease governance-policy-framework in namespace klusterlet-2sti77pkq7agiit4e9p66ic3479hnrq5 on cluster ci01-j1089664-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-18T18:05:44Z
  Severity:    Sev3
  Labels:      alertname="LeaderElectionLeaseStale", cluster="ci01-j1089664-mgmt-2", component="leader-election", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.54:8080", job="kube-state-metrics", lease="config-policy-controller", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1089664/providers/microsoft.monitor/accounts/services-j1089664", namespace="klusterlet-2sti77pkq7agiit4e9p66ic3479hnrq5", pod="arohcp-monitor-kube-state-metrics-6486647766-6vsrg", prometheus="prometheus/prometheus", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Leader election lease config-policy-controller in namespace klusterlet-2sti77pkq7agiit4e9p66ic3479hnrq5 on cluster ci01-j1089664-mgmt-2 has not been renewed for more than 37 minutes. The leadership election might be broken or the component stopped running.

time=2026-09-18T15:40:00.669Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring description="Step monitoring\n  Kind: ARMStack\n  Template: templates/monitoring.bicep\n  Parameters: configurations/monitoring.tmpl.bicepparam"
time=2026-09-18T15:42:51.615Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Monitoring resourceGroup=regional step=monitoring err="failed to run ARM step: failed to wait for deployment stack at resource group scope: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/eastus2/deploymentStackOperationStatus/326eddf6-9965-4299-86de-7e1eb754418c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentStackDeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/locations/eastus2/deploymentStackOperationStatus/326eddf6-9965-4299-86de-7e1eb754418c\",\n  \"name\": \"326eddf6-9965-4299-86de-7e1eb754418c\",\n  \"status\": \"failed\",\n  \"error\": {\n    \"code\": \"DeploymentStackDeploymentFailed\",\n    \"message\": \"One or more resources could not be deployed. Correlation id: 'e78fa208-8c43-45b6-a80c-57ad93c6a044'.\",\n    \"details\": [\n      {\n        \"code\": \"DeploymentFailed\",\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j4742016/providers/Microsoft.Resources/deployments/ka7tdccaiavlrtibmy3y5s7tfhcs6gpzxc6p4xe5pchjy-26091815bptz8\",\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n        \"details\": [\n          {\n            \"code\": \"BadRequest\",\n            \"message\": \"Property name: Rules[0].Expression, Attempted value: null, Error: 1:237: parse error: unknown escape sequence U+002E '.' Activity ID: d9b78f81-a4e0-4b15-9e4d-75e6f5c342e3.\"\n          }\n        ]\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T14:32:08Z
  Ended:       2026-09-18T14:39:08Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1716224-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1716224-svc).

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001520108>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-ptc9sq59cf2d" to finish deleting: GET https://rp.j6053632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3852ce7c-96ee-4d91-9970-a39897a9c07a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3852ce7c-96ee-4d91-9970-a39897a9c07a",
      "name": "3852ce7c-96ee-4d91-9970-a39897a9c07a",
      "status": "Failed",
      "startTime": "2026-09-18T14:15:14.231768617Z",
      "endTime": "2026-09-18T14:39:19.156773648Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2stfc33bsiqt6ka6nm1u076ksjbrgtab still exists (deletion dispatched at 2026-09-18T14:15:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001520108>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="delayed-rbac-cluster" in resourcegroup="delayed-rbac-ptc9sq59cf2d" to finish deleting: GET https://rp.j6053632.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3852ce7c-96ee-4d91-9970-a39897a9c07a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/3852ce7c-96ee-4d91-9970-a39897a9c07a",
      "name": "3852ce7c-96ee-4d91-9970-a39897a9c07a",
      "status": "Failed",
      "startTime": "2026-09-18T14:15:14.231768617Z",
      "endTime": "2026-09-18T14:39:19.156773648Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2stfc33bsiqt6ka6nm1u076ksjbrgtab still exists (deletion dispatched at 2026-09-18T14:15:39Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T13:28:52Z
  Ended:       2026-09-18T13:38:52Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j6576768-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters exceeded 1 second over the past 30 minutes (cluster ci01-j6576768-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:115]: failed to verify simple web app runs on cluster "cluster-hs-pre-q78vhx"
Unexpected error:
    <*fmt.wrapError | 0xc000f621e0>: 
    DNS for route host agnhost-e2e-sample-app-s69wn.apps.aro.o5k6a1c8v4j7o5i.oz3q.j2056320.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-s69wn.apps.aro.o5k6a1c8v4j7o5i.oz3q.j2056320.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-s69wn.apps.aro.o5k6a1c8v4j7o5i.oz3q.j2056320.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:115]: failed to verify simple web app runs on cluster "cluster-hs-pre-q78vhx"
Unexpected error:
    <*fmt.wrapError | 0xc000f621e0>: 
    DNS for route host agnhost-e2e-sample-app-s69wn.apps.aro.o5k6a1c8v4j7o5i.oz3q.j2056320.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-s69wn.apps.aro.o5k6a1c8v4j7o5i.oz3q.j2056320.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-s69wn.apps.aro.o5k6a1c8v4j7o5i.oz3q.j2056320.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc0015787a0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-7x7svb59cvl9" to finish creating: GET https://rp.j2056320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/79d978b9-66a2-443c-bcf8-f3536818238d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/79d978b9-66a2-443c-bcf8-f3536818238d",
      "name": "79d978b9-66a2-443c-bcf8-f3536818238d",
      "status": "Failed",
      "startTime": "2026-09-18T12:55:05.057839777Z",
      "endTime": "2026-09-18T13:24:10.88915368Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m15s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc0015787a0>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-7x7svb59cvl9" to finish creating: GET https://rp.j2056320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/79d978b9-66a2-443c-bcf8-f3536818238d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/79d978b9-66a2-443c-bcf8-f3536818238d",
      "name": "79d978b9-66a2-443c-bcf8-f3536818238d",
      "status": "Failed",
      "startTime": "2026-09-18T12:55:05.057839777Z",
      "endTime": "2026-09-18T13:24:10.88915368Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m15s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000fae660>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-77t5pbg7l7p6" to finish creating: GET https://rp.j2056320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9382644d-58f7-4d28-9aa1-a5cbc03fae7d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9382644d-58f7-4d28-9aa1-a5cbc03fae7d",
      "name": "9382644d-58f7-4d28-9aa1-a5cbc03fae7d",
      "status": "Failed",
      "startTime": "2026-09-18T12:53:37.696598816Z",
      "endTime": "2026-09-18T13:22:40.932765907Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m26s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000fae660>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-77t5pbg7l7p6" to finish creating: GET https://rp.j2056320.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9382644d-58f7-4d28-9aa1-a5cbc03fae7d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9382644d-58f7-4d28-9aa1-a5cbc03fae7d",
      "name": "9382644d-58f7-4d28-9aa1-a5cbc03fae7d",
      "status": "Failed",
      "startTime": "2026-09-18T12:53:37.696598816Z",
      "endTime": "2026-09-18T13:22:40.932765907Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m26s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 3 time(s) (2 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T13:05:51Z
  Ended:       2026-09-18T13:34:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2056320-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2steckt12k3f6paulmdgtiackkj5k84r-j7s3j8r2u9z3h1j", pod="router-86c7fcbd7d-pjgqb", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2steckt12k3f6paulmdgtiackkj5k84r-j7s3j8r2u9z3h1j/router-86c7fcbd7d-pjgqb has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-18T13:07:43Z
  Ended:       2026-09-18T13:35:44Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2056320-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2stedamranrq9pqrbs02e1rt645clovm-cilium-cluster", pod="router-7c5f44b8f5-2rxm6", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2stedamranrq9pqrbs02e1rt645clovm-cilium-cluster/router-7c5f44b8f5-2rxm6 has been in a non-ready state for longer than 5 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T13:14:45Z
  Ended:       2026-09-18T13:44:43Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2056320-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2056320/providers/microsoft.monitor/accounts/hcps-j2056320", namespace="ocm-arohcpci01-2stecjn5ajol8jdm27uamrva75b6nf2v-r5f9v1s9k2k3j8y", pod="arohcp-monitor-kube-state-metrics-6486647766-gdfgt", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2stecjn5ajol8jdm27uamrva75b6nf2v-r5f9v1s9k2k3j8y/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-18T13:14:45Z
  Ended:       2026-09-18T13:45:42Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2056320-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2056320/providers/microsoft.monitor/accounts/hcps-j2056320", namespace="ocm-arohcpci01-2stecjn5ajol8jdm27uamrva75b6nf2v-r5f9v1s9k2k3j8y", pod="arohcp-monitor-kube-state-metrics-6486647766-gdfgt", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2stecjn5ajol8jdm27uamrva75b6nf2v-r5f9v1s9k2k3j8y/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T13:16:41Z
  Ended:       2026-09-18T13:44:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2056320-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2056320/providers/microsoft.monitor/accounts/hcps-j2056320", namespace="ocm-arohcpci01-2stecjn5ajol8jdm27uamrva75b6nf2v-r5f9v1s9k2k3j8y", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="af416a40-e08e-480e-bf5b-da79914398be"
  Description: Pod ocm-arohcpci01-2stecjn5ajol8jdm27uamrva75b6nf2v-r5f9v1s9k2k3j8y/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-18T13:17:41Z
  Ended:       2026-09-18T13:45:39Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2056320-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2056320/providers/microsoft.monitor/accounts/hcps-j2056320", namespace="ocm-arohcpci01-2stecjn5ajol8jdm27uamrva75b6nf2v-r5f9v1s9k2k3j8y", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="af416a40-e08e-480e-bf5b-da79914398be"
  Description: Pod ocm-arohcpci01-2stecjn5ajol8jdm27uamrva75b6nf2v-r5f9v1s9k2k3j8y/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T13:28:41Z
  Ended:       2026-09-18T13:42:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2056320-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2056320/providers/microsoft.monitor/accounts/hcps-j2056320", namespace="ocm-arohcpci01-2stecjn5ajol8jdm27uamrva75b6nf2v-r5f9v1s9k2k3j8y", pod="arohcp-monitor-kube-state-metrics-6486647766-gdfgt", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2stecjn5ajol8jdm27uamrva75b6nf2v-r5f9v1s9k2k3j8y/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-18T13:28:41Z
  Ended:       2026-09-18T13:42:41Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j2056320-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2056320/providers/microsoft.monitor/accounts/hcps-j2056320", namespace="ocm-arohcpci01-2stecjn5ajol8jdm27uamrva75b6nf2v-r5f9v1s9k2k3j8y", pod="arohcp-monitor-kube-state-metrics-6486647766-gdfgt", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2stecjn5ajol8jdm27uamrva75b6nf2v-r5f9v1s9k2k3j8y/capi-provider has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T11:02:41Z
  Ended:       2026-09-18T11:23:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5658496-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2stcj8snr7hlhv8hfthg1emt8dhhfded-j1a6w4y5x8t0x1d", pod="router-769f87966d-khp89", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2stcj8snr7hlhv8hfthg1emt8dhhfded-j1a6w4y5x8t0x1d/router-769f87966d-khp89 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-18T11:04:44Z
  Ended:       2026-09-18T11:23:40Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5658496-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2stckihvk61lv7eojs5hbnlalilj7v1i-agg-cluster", pod="router-7c9ffb6845-wczwq", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2stckihvk61lv7eojs5hbnlalilj7v1i-agg-cluster/router-7c9ffb6845-wczwq has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T11:09:41Z
  Ended:       2026-09-18T11:20:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9620352-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2stclp139sl99taqsqoij7cie2hd4cpk-v20260901", pod="router-5b84bc6565-scjwf", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2stclp139sl99taqsqoij7cie2hd4cpk-v20260901/router-5b84bc6565-scjwf has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T11:21:18Z
  Ended:       2026-09-18T11:37:18Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3790208-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3790208-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T11:14:28Z
  Ended:       2026-09-18T11:23:28Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3790208-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2stcn6svivskocqagj4qaaqujiq0cekp-m1e6y0x7h1c1n8p", pod="router-748fddb496-fcbbx", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2stcn6svivskocqagj4qaaqujiq0cekp-m1e6y0x7h1c1n8p/router-748fddb496-fcbbx has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc001084500>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-95vjt5mdwdlk" to finish creating: GET https://rp.j8771328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ec69745-9888-4c12-ad3c-cfe984e0f2d3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ec69745-9888-4c12-ad3c-cfe984e0f2d3",
      "name": "0ec69745-9888-4c12-ad3c-cfe984e0f2d3",
      "status": "Failed",
      "startTime": "2026-09-18T10:07:19.283105586Z",
      "endTime": "2026-09-18T10:36:28.169644011Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc001084500>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-95vjt5mdwdlk" to finish creating: GET https://rp.j8771328.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ec69745-9888-4c12-ad3c-cfe984e0f2d3
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0ec69745-9888-4c12-ad3c-cfe984e0f2d3",
      "name": "0ec69745-9888-4c12-ad3c-cfe984e0f2d3",
      "status": "Failed",
      "startTime": "2026-09-18T10:07:19.283105586Z",
      "endTime": "2026-09-18T10:36:28.169644011Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s) (1 unknown, 1 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T10:18:52Z
  Ended:       2026-09-18T10:30:58Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8771328-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2stbuceofnif99sq6etloedbqkpc5qh9-u1a3e7t8x9z9b1l", pod="router-b6965d7dc-jzrb7", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2stbuceofnif99sq6etloedbqkpc5qh9-u1a3e7t8x9z9b1l/router-b6965d7dc-jzrb7 has been in a non-ready state for longer than 5 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T10:27:52Z
  Ended:       2026-09-18T10:54:52Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8771328-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.161:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8771328/providers/microsoft.monitor/accounts/hcps-j8771328", namespace="ocm-arohcpci01-2stbulpvmbabhuosdje4290vg8cio72d-j5i4m0f4c4k3j8e", pod="arohcp-monitor-kube-state-metrics-574f657d96-nsf7x", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2stbulpvmbabhuosdje4290vg8cio72d-j5i4m0f4c4k3j8e/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-18T10:28:56Z
  Ended:       2026-09-18T10:54:52Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8771328-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.161:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8771328/providers/microsoft.monitor/accounts/hcps-j8771328", namespace="ocm-arohcpci01-2stbulpvmbabhuosdje4290vg8cio72d-j5i4m0f4c4k3j8e", pod="arohcp-monitor-kube-state-metrics-574f657d96-nsf7x", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2stbulpvmbabhuosdje4290vg8cio72d-j5i4m0f4c4k3j8e/etcd has not matched the expected number of replicas for longer than 15 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-l7rx685m27vp/cluster-candidate-4-20-4l4f7q should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f9ac60>: 
    failed to create HCP cluster cluster-candidate-4-20-4l4f7q: failed waiting for cluster="cluster-candidate-4-20-4l4f7q" in resourcegroup="rg-candidate-4-20-l7rx685m27vp" to finish creating: GET https://rp.j3745536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d0d44620-2f5c-46e9-97e5-7104c5ac9593
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d0d44620-2f5c-46e9-97e5-7104c5ac9593",
      "name": "d0d44620-2f5c-46e9-97e5-7104c5ac9593",
      "status": "Failed",
      "startTime": "2026-09-18T09:24:34.52165977Z",
      "endTime": "2026-09-18T09:53:41.437046134Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 21m0s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-l7rx685m27vp/cluster-candidate-4-20-4l4f7q should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f9ac60>: 
    failed to create HCP cluster cluster-candidate-4-20-4l4f7q: failed waiting for cluster="cluster-candidate-4-20-4l4f7q" in resourcegroup="rg-candidate-4-20-l7rx685m27vp" to finish creating: GET https://rp.j3745536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d0d44620-2f5c-46e9-97e5-7104c5ac9593
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d0d44620-2f5c-46e9-97e5-7104c5ac9593",
      "name": "d0d44620-2f5c-46e9-97e5-7104c5ac9593",
      "status": "Failed",
      "startTime": "2026-09-18T09:24:34.52165977Z",
      "endTime": "2026-09-18T09:53:41.437046134Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.20.38 is Partial (want Completed), started 21m0s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000faea40>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-tdl2drfvfhtk" to finish creating: GET https://rp.j3745536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4fc5c348-7117-4850-997b-20eb599df30e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4fc5c348-7117-4850-997b-20eb599df30e",
      "name": "4fc5c348-7117-4850-997b-20eb599df30e",
      "status": "Failed",
      "startTime": "2026-09-18T09:25:07.592750123Z",
      "endTime": "2026-09-18T09:54:11.523791716Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 21m55s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000faea40>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-tdl2drfvfhtk" to finish creating: GET https://rp.j3745536.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4fc5c348-7117-4850-997b-20eb599df30e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4fc5c348-7117-4850-997b-20eb599df30e",
      "name": "4fc5c348-7117-4850-997b-20eb599df30e",
      "status": "Failed",
      "startTime": "2026-09-18T09:25:07.592750123Z",
      "endTime": "2026-09-18T09:54:11.523791716Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 21m55s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.002s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T09:36:00Z
  Ended:       2026-09-18T10:18:57Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3745536-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2stba81rimklhsg3cr5ktr5tf749cko1-u3d9c8l0y5d3t2z", pod="router-5b649bbbf7-4cvr4", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2stba81rimklhsg3cr5ktr5tf749cko1-u3d9c8l0y5d3t2z/router-5b649bbbf7-4cvr4 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000adcaa0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-g5jssdfr4s6w" to finish creating: GET https://rp.j5368576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/92cf1c4a-bb92-46ae-81a1-0fe741329a13
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/92cf1c4a-bb92-46ae-81a1-0fe741329a13",
      "name": "92cf1c4a-bb92-46ae-81a1-0fe741329a13",
      "status": "Failed",
      "startTime": "2026-09-18T09:29:20.728673075Z",
      "endTime": "2026-09-18T09:58:22.849626679Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m41s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000adcaa0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-g5jssdfr4s6w" to finish creating: GET https://rp.j5368576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/92cf1c4a-bb92-46ae-81a1-0fe741329a13
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/92cf1c4a-bb92-46ae-81a1-0fe741329a13",
      "name": "92cf1c4a-bb92-46ae-81a1-0fe741329a13",
      "status": "Failed",
      "startTime": "2026-09-18T09:29:20.728673075Z",
      "endTime": "2026-09-18T09:58:22.849626679Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m41s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000f7e720>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-8kv74t8cs9l5" to finish creating: GET https://rp.j5368576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4ec06138-7d7f-447e-acf1-59e926e58e3b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4ec06138-7d7f-447e-acf1-59e926e58e3b",
      "name": "4ec06138-7d7f-447e-acf1-59e926e58e3b",
      "status": "Failed",
      "startTime": "2026-09-18T09:29:55.514391889Z",
      "endTime": "2026-09-18T09:59:02.761026905Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 23m13s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000f7e720>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-8kv74t8cs9l5" to finish creating: GET https://rp.j5368576.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4ec06138-7d7f-447e-acf1-59e926e58e3b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4ec06138-7d7f-447e-acf1-59e926e58e3b",
      "name": "4ec06138-7d7f-447e-acf1-59e926e58e3b",
      "status": "Failed",
      "startTime": "2026-09-18T09:29:55.514391889Z",
      "endTime": "2026-09-18T09:59:02.761026905Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 23m13s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T09:41:54Z
  Ended:       2026-09-18T10:11:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5368576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2stbcs84sip93t6dffm9229aerut2acv-z8r9c8z3p9s9m1i", pod="router-d76b55d7-rgg2v", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2stbcs84sip93t6dffm9229aerut2acv-z8r9c8z3p9s9m1i/router-d76b55d7-rgg2v has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-18T09:42:48Z
  Ended:       2026-09-18T10:11:52Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j5368576-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2stbd564riaqku802qrnf0q9bc8r9vik-private-kas", pod="router-85dc58d9b7-tsdrp", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2stbd564riaqku802qrnf0q9bc8r9vik-private-kas/router-85dc58d9b7-tsdrp has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0031a8160>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-p9htbjsmj226" to finish creating: GET https://rp.j6818048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1616a745-fc49-408c-a84f-41679708fa49
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1616a745-fc49-408c-a84f-41679708fa49",
      "name": "1616a745-fc49-408c-a84f-41679708fa49",
      "status": "Failed",
      "startTime": "2026-09-18T09:27:23.7095894Z",
      "endTime": "2026-09-18T09:28:03.46142651Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0031a8160>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-p9htbjsmj226" to finish creating: GET https://rp.j6818048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1616a745-fc49-408c-a84f-41679708fa49
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1616a745-fc49-408c-a84f-41679708fa49",
      "name": "1616a745-fc49-408c-a84f-41679708fa49",
      "status": "Failed",
      "startTime": "2026-09-18T09:27:23.7095894Z",
      "endTime": "2026-09-18T09:28:03.46142651Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:383]: Timed out after 600.008s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc000909420>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:383]: Timed out after 600.008s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc000909420>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000a9a600>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-pwncww98jzm8" to finish creating: GET https://rp.j6818048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/145961c0-6282-4bf7-a47e-98a2a7475ba8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/145961c0-6282-4bf7-a47e-98a2a7475ba8",
      "name": "145961c0-6282-4bf7-a47e-98a2a7475ba8",
      "status": "Failed",
      "startTime": "2026-09-18T09:11:52.011321272Z",
      "endTime": "2026-09-18T09:40:57.964346748Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m56s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000a9a600>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-pwncww98jzm8" to finish creating: GET https://rp.j6818048.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/145961c0-6282-4bf7-a47e-98a2a7475ba8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/145961c0-6282-4bf7-a47e-98a2a7475ba8",
      "name": "145961c0-6282-4bf7-a47e-98a2a7475ba8",
      "status": "Failed",
      "startTime": "2026-09-18T09:11:52.011321272Z",
      "endTime": "2026-09-18T09:40:57.964346748Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 23m56s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T09:27:59Z
  Ended:       2026-09-18T09:34:55Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6818048-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 7 time(s) (1 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T09:24:01Z
  Ended:       2026-09-18T09:49:00Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6818048-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2stb4ma2bs7029nmfb05hiqpcdsgabul-w1y0u6t1v1n3z9l", pod="router-6bd77b545b-pvt7r", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2stb4ma2bs7029nmfb05hiqpcdsgabul-w1y0u6t1v1n3z9l/router-6bd77b545b-pvt7r has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T09:23:28Z
  Ended:       2026-09-18T09:39:34Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j8030592-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc001a0ed40>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:59914->172.30.0.10:53: read: connection refused
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc001a0ed40>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:59914->172.30.0.10:53: read: connection refused
    ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T08:31:06Z
  Ended:       2026-09-18T08:38:05Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j5294976-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j5294976-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T08:33:35Z
  Ended:       2026-09-18T08:39:36Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j4018048-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j4018048-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T08:44:25Z
  Ended:       2026-09-18T08:50:22Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j4018048-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sta451luu6i39gfi3s8vomm21o2b1lj-v1u6q6s5g4j0h3p", pod="kube-apiserver-5959d9c69f-k9pqp", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sta451luu6i39gfi3s8vomm21o2b1lj-v1u6q6s5g4j0h3p/kube-apiserver-5959d9c69f-k9pqp has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.37
Unexpected error:
    <*fmt.wrapError | 0xc00100a760>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-ll6m5x" in resourcegroup="rg-np-version-upgrade-ll6m5x-2g2kcltq9hpv" to finish creating: GET https://rp.j5412736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a7e813a9-2095-43ef-9c86-073d007f6602
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a7e813a9-2095-43ef-9c86-073d007f6602",
      "name": "a7e813a9-2095-43ef-9c86-073d007f6602",
      "status": "Failed",
      "startTime": "2026-09-18T07:07:49.634853409Z",
      "endTime": "2026-09-18T07:26:53.294963336Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.37
Unexpected error:
    <*fmt.wrapError | 0xc00100a760>: 
    failed to create NodePool npupgrade-4-20: failed waiting for nodepool="npupgrade-4-20" for cluster "np-version-upgrade-cluster-ll6m5x" in resourcegroup="rg-np-version-upgrade-ll6m5x-2g2kcltq9hpv" to finish creating: GET https://rp.j5412736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a7e813a9-2095-43ef-9c86-073d007f6602
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a7e813a9-2095-43ef-9c86-073d007f6602",
      "name": "a7e813a9-2095-43ef-9c86-073d007f6602",
      "status": "Failed",
      "startTime": "2026-09-18T07:07:49.634853409Z",
      "endTime": "2026-09-18T07:26:53.294963336Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-895tpwlfvzpg/cluster-candidate-5-0-jpmgmm should provision
Unexpected error:
    <*fmt.wrapError | 0xc00015e760>: 
    failed to create HCP cluster cluster-candidate-5-0-jpmgmm: failed waiting for cluster="cluster-candidate-5-0-jpmgmm" in resourcegroup="rg-candidate-5-0-895tpwlfvzpg" to finish creating: GET https://rp.j5412736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d838caae-4be9-4361-b2b6-0820103db4f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d838caae-4be9-4361-b2b6-0820103db4f9",
      "name": "d838caae-4be9-4361-b2b6-0820103db4f9",
      "status": "Failed",
      "startTime": "2026-09-18T06:57:19.894043957Z",
      "endTime": "2026-09-18T07:26:23.420290614Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 21m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-895tpwlfvzpg/cluster-candidate-5-0-jpmgmm should provision
Unexpected error:
    <*fmt.wrapError | 0xc00015e760>: 
    failed to create HCP cluster cluster-candidate-5-0-jpmgmm: failed waiting for cluster="cluster-candidate-5-0-jpmgmm" in resourcegroup="rg-candidate-5-0-895tpwlfvzpg" to finish creating: GET https://rp.j5412736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d838caae-4be9-4361-b2b6-0820103db4f9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/d838caae-4be9-4361-b2b6-0820103db4f9",
      "name": "d838caae-4be9-4361-b2b6-0820103db4f9",
      "status": "Failed",
      "startTime": "2026-09-18T06:57:19.894043957Z",
      "endTime": "2026-09-18T07:26:23.420290614Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 21m59s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001c80150>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-nglkjctcmb9p" to finish deleting: GET https://rp.j5412736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9724272a-25ec-4c0a-b43f-3f85230b0b68
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9724272a-25ec-4c0a-b43f-3f85230b0b68",
      "name": "9724272a-25ec-4c0a-b43f-3f85230b0b68",
      "status": "Failed",
      "startTime": "2026-09-18T07:10:06.620770558Z",
      "endTime": "2026-09-18T07:34:13.53272476Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2st95o86ftkou4io40i8o12g13uerlpd still exists (deletion dispatched at 2026-09-18T07:10:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc001c80150>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-nglkjctcmb9p" to finish deleting: GET https://rp.j5412736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9724272a-25ec-4c0a-b43f-3f85230b0b68
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/9724272a-25ec-4c0a-b43f-3f85230b0b68",
      "name": "9724272a-25ec-4c0a-b43f-3f85230b0b68",
      "status": "Failed",
      "startTime": "2026-09-18T07:10:06.620770558Z",
      "endTime": "2026-09-18T07:34:13.53272476Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2st95o86ftkou4io40i8o12g13uerlpd still exists (deletion dispatched at 2026-09-18T07:10:37Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00080af80>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-4pnfzqpsrckd" to finish creating: GET https://rp.j5412736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/540f69ae-a13e-46ee-bf0d-4de42d2cfa88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/540f69ae-a13e-46ee-bf0d-4de42d2cfa88",
      "name": "540f69ae-a13e-46ee-bf0d-4de42d2cfa88",
      "status": "Failed",
      "startTime": "2026-09-18T06:58:33.125357484Z",
      "endTime": "2026-09-18T07:27:33.397040852Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m38s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00080af80>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-4pnfzqpsrckd" to finish creating: GET https://rp.j5412736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/540f69ae-a13e-46ee-bf0d-4de42d2cfa88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/540f69ae-a13e-46ee-bf0d-4de42d2cfa88",
      "name": "540f69ae-a13e-46ee-bf0d-4de42d2cfa88",
      "status": "Failed",
      "startTime": "2026-09-18T06:58:33.125357484Z",
      "endTime": "2026-09-18T07:27:33.397040852Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m38s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019d60f0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-vmz65bf72428" to finish deleting: GET https://rp.j5412736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e838e2dd-c8ac-447c-b2c0-095cbbe04623
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e838e2dd-c8ac-447c-b2c0-095cbbe04623",
      "name": "e838e2dd-c8ac-447c-b2c0-095cbbe04623",
      "status": "Failed",
      "startTime": "2026-09-18T07:07:58.803605122Z",
      "endTime": "2026-09-18T07:32:03.445509106Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2st95msfbb4nhua5a6pq8m3044dhe0se still exists (deletion dispatched at 2026-09-18T07:08:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc0019d60f0>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="patch-tags-cluster" in resourcegroup="patch-tags-vmz65bf72428" to finish deleting: GET https://rp.j5412736.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e838e2dd-c8ac-447c-b2c0-095cbbe04623
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/e838e2dd-c8ac-447c-b2c0-095cbbe04623",
      "name": "e838e2dd-c8ac-447c-b2c0-095cbbe04623",
      "status": "Failed",
      "startTime": "2026-09-18T07:07:58.803605122Z",
      "endTime": "2026-09-18T07:32:03.445509106Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2st95msfbb4nhua5a6pq8m3044dhe0se still exists (deletion dispatched at 2026-09-18T07:08:00Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [hostedCluster] HostedCluster still exists; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.001s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.001s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: ...
to be true

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.002s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc000c8c390>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:35668->172.30.0.10:53: read: connection refused
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.002s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc000c8c390>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.10:35668->172.30.0.10:53: read: connection refused
    ...

alert fired 9 time(s) (3 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T06:59:52Z
  Ended:       2026-09-18T07:06:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7100672-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st91kv9p4h3i5f2qjatf1ddh7hg451l-u6e3o9h6n4t0q1m", pod="router-56dbc9cf68-flvzm", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st91kv9p4h3i5f2qjatf1ddh7hg451l-u6e3o9h6n4t0q1m/router-56dbc9cf68-flvzm has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-18T07:00:48Z
  Ended:       2026-09-18T07:18:51Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7100672-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st91ihbjq7vgovat1v96s2dgh6gc8k8-ingress-cert-le", pod="router-866f65df77-vw5dl", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st91ihbjq7vgovat1v96s2dgh6gc8k8-ingress-cert-le/router-866f65df77-vw5dl has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-18T07:00:48Z
  Ended:       2026-09-18T07:06:48Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j7100672-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st91vs1vhcp81dngelnmuif63cd1p56-private-kas", pod="router-6449dfbd8c-5w442", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st91vs1vhcp81dngelnmuif63cd1p56-private-kas/router-6449dfbd8c-5w442 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc0013f8e20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-9ntdl6mdb8xb" to finish creating: GET https://rp.j8102400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f794a701-689e-4889-b351-a1feb52109b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f794a701-689e-4889-b351-a1feb52109b7",
      "name": "f794a701-689e-4889-b351-a1feb52109b7",
      "status": "Failed",
      "startTime": "2026-09-18T06:13:08.560808434Z",
      "endTime": "2026-09-18T06:32:13.567780508Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc0013f8e20>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-9ntdl6mdb8xb" to finish creating: GET https://rp.j8102400.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f794a701-689e-4889-b351-a1feb52109b7
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f794a701-689e-4889-b351-a1feb52109b7",
      "name": "f794a701-689e-4889-b351-a1feb52109b7",
      "status": "Failed",
      "startTime": "2026-09-18T06:13:08.560808434Z",
      "endTime": "2026-09-18T06:32:13.567780508Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:403]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc0009cb668>: 
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-6k7xjqv9k7sk" to finish creating: GET https://rp.j1065216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6b3fe34c-12d6-487f-9607-688763ac04db
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6b3fe34c-12d6-487f-9607-688763ac04db",
      "name": "6b3fe34c-12d6-487f-9607-688763ac04db",
      "status": "Failed",
      "startTime": "2026-09-18T06:12:51.736703148Z",
      "endTime": "2026-09-18T06:31:55.141914948Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:403]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc0009cb668>: 
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-6k7xjqv9k7sk" to finish creating: GET https://rp.j1065216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6b3fe34c-12d6-487f-9607-688763ac04db
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6b3fe34c-12d6-487f-9607-688763ac04db",
      "name": "6b3fe34c-12d6-487f-9607-688763ac04db",
      "status": "Failed",
      "startTime": "2026-09-18T06:12:51.736703148Z",
      "endTime": "2026-09-18T06:31:55.141914948Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.009s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: ...
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:159]: Timed out after 660.009s.
at least one schedule should have a LastBackupTime
Expected
    <bool>: ...
to be true

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000a46a00>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-jt9b8z9tj78s" to finish creating: GET https://rp.j0748928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/16ca80cc-2a38-4172-a9cc-5c6829de0ae2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/16ca80cc-2a38-4172-a9cc-5c6829de0ae2",
      "name": "16ca80cc-2a38-4172-a9cc-5c6829de0ae2",
      "status": "Failed",
      "startTime": "2026-09-18T06:13:10.341418654Z",
      "endTime": "2026-09-18T06:42:12.545110519Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000a46a00>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-jt9b8z9tj78s" to finish creating: GET https://rp.j0748928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/16ca80cc-2a38-4172-a9cc-5c6829de0ae2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/16ca80cc-2a38-4172-a9cc-5c6829de0ae2",
      "name": "16ca80cc-2a38-4172-a9cc-5c6829de0ae2",
      "status": "Failed",
      "startTime": "2026-09-18T06:13:10.341418654Z",
      "endTime": "2026-09-18T06:42:12.545110519Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d7ccc0>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-xp4rlvdpbgj6" to finish creating: GET https://rp.j0748928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b344e921-2b47-4118-aac7-81c45cb67cc0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b344e921-2b47-4118-aac7-81c45cb67cc0",
      "name": "b344e921-2b47-4118-aac7-81c45cb67cc0",
      "status": "Failed",
      "startTime": "2026-09-18T06:13:19.863238771Z",
      "endTime": "2026-09-18T06:42:22.545828006Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d7ccc0>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-xp4rlvdpbgj6" to finish creating: GET https://rp.j0748928.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b344e921-2b47-4118-aac7-81c45cb67cc0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b344e921-2b47-4118-aac7-81c45cb67cc0",
      "name": "b344e921-2b47-4118-aac7-81c45cb67cc0",
      "status": "Failed",
      "startTime": "2026-09-18T06:13:19.863238771Z",
      "endTime": "2026-09-18T06:42:22.545828006Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:139]: failed to get backup schedules for cadence check
Unexpected error:
    <*errors.errorString | 0xc0010c3e40>: 
    expected status 200, got 412: {
        "error": {
            "code": "InvalidResource",
            "message": "kube-applier client not available for management cluster /providers/microsoft.redhatopenshift/stamps/1/managementclusters/default"
        }
    }
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:139]: failed to get backup schedules for cadence check
Unexpected error:
    <*errors.errorString | 0xc0010c3e40>: 
    expected status 200, got 412: {
        "error": {
            "code": "InvalidResource",
            "message": "kube-applier client not available for management cluster /providers/microsoft.redhatopenshift/stamps/1/managementclusters/default"
        }
    }
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T06:22:09Z
  Ended:       2026-09-18T06:28:10Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0748928-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0748928-svc).

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-18T07:08:32Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j0748928-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-9hl75", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-9hl75 (namespace arobit) on cluster ci01-j0748928-svc has averaged 125.7% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 7 time(s) (1 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T06:00:50Z
  Ended:       2026-09-18T06:20:50Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2458624-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st858a4pvmlmac4uvehhfkife65pian-u9l5q3l8s3c3d7s", pod="router-d9f7df5f5-npvv8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st858a4pvmlmac4uvehhfkife65pian-u9l5q3l8s3c3d7s/router-d9f7df5f5-npvv8 has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--xq7n5gbztqbj/cluster-hs-pre-kf6554 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001090320>: 
    failed to create HCP cluster cluster-hs-pre-kf6554: failed waiting for cluster="cluster-hs-pre-kf6554" in resourcegroup="rg-hs-pre--xq7n5gbztqbj" to finish creating: GET https://rp.j2362240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/359ebfce-873d-4d6c-8f64-6e12fb114150
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/359ebfce-873d-4d6c-8f64-6e12fb114150",
      "name": "359ebfce-873d-4d6c-8f64-6e12fb114150",
      "status": "Failed",
      "startTime": "2026-09-18T04:02:43.525962262Z",
      "endTime": "2026-09-18T04:31:43.022568987Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--xq7n5gbztqbj/cluster-hs-pre-kf6554 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001090320>: 
    failed to create HCP cluster cluster-hs-pre-kf6554: failed waiting for cluster="cluster-hs-pre-kf6554" in resourcegroup="rg-hs-pre--xq7n5gbztqbj" to finish creating: GET https://rp.j2362240.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/359ebfce-873d-4d6c-8f64-6e12fb114150
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/359ebfce-873d-4d6c-8f64-6e12fb114150",
      "name": "359ebfce-873d-4d6c-8f64-6e12fb114150",
      "status": "Failed",
      "startTime": "2026-09-18T04:02:43.525962262Z",
      "endTime": "2026-09-18T04:31:43.022568987Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T04:31:05Z
  Ended:       2026-09-18T04:48:04Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j2362240-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2362240/providers/microsoft.monitor/accounts/hcps-j2362240", namespace="ocm-arohcpci01-2st6jp6ln5eij992a4h7v8fe6stngjn4-s9g6g5z2i0s8b2f", pod="etcd-1", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="e333cab1-7b1b-43d4-a2a8-c38ce512c9fe"
  Description: Pod ocm-arohcpci01-2st6jp6ln5eij992a4h7v8fe6stngjn4-s9g6g5z2i0s8b2f/etcd-1 (etcd) is in waiting state (reason: "CrashLoopBackOff").

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T04:18:48Z
  Ended:       2026-09-18T04:37:50Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j2362240-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T04:29:08Z
  Ended:       2026-09-18T04:50:06Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2362240-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2362240/providers/microsoft.monitor/accounts/hcps-j2362240", namespace="ocm-arohcpci01-2st6jp6ln5eij992a4h7v8fe6stngjn4-s9g6g5z2i0s8b2f", pod="arohcp-monitor-kube-state-metrics-574f657d96-2r9f8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2st6jp6ln5eij992a4h7v8fe6stngjn4-s9g6g5z2i0s8b2f/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-18T04:29:08Z
  Ended:       2026-09-18T04:55:05Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j2362240-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.78:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j2362240/providers/microsoft.monitor/accounts/hcps-j2362240", namespace="ocm-arohcpci01-2st6jp6ln5eij992a4h7v8fe6stngjn4-s9g6g5z2i0s8b2f", pod="arohcp-monitor-kube-state-metrics-574f657d96-2r9f8", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2st6jp6ln5eij992a4h7v8fe6stngjn4-s9g6g5z2i0s8b2f/etcd has not matched the expected number of replicas for longer than 15 minutes.

time=2026-09-18T03:37:37.864Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-18T03:37:38.393Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1
time=2026-09-18T03:37:39.053Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 deployment=9d03ef43fb389999fc408f90bd02e6c94ce2721f8e17a8400c8a777195fbc298 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j1401344-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F9d03ef43fb389999fc408f90bd02e6c94ce2721f8e17a8400c8a777195fbc298
time=2026-09-18T03:40:39.648Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1401344-mgmt-1/providers/Microsoft.Resources/deployments/9d03ef43fb389999fc408f90bd02e6c94ce2721f8e17a8400c8a777195fbc298/operationStatuses/08584119042267536349\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1401344-mgmt-1/providers/Microsoft.Resources/deployments/cluster-sr7huchijtjyg\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"BadRequest\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j1401344-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j1401344-mgmt-1\\\",\\r\\n            \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"GetMSICredentialError\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Get managed identity credential failed. Details: Category: ClientError; Code: NotFound; SubCode: NotFound; Message: unexpected response from MSI data plane, length of returned certificate: 0; Dependency: MSI Credential; AKSTeam: RP\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-8rxkwjmg9fdn/cluster-candidate-4-22-8z2p5b should provision
Unexpected error:
    <*fmt.wrapError | 0xc0017682a0>: 
    failed to create HCP cluster cluster-candidate-4-22-8z2p5b: failed waiting for cluster="cluster-candidate-4-22-8z2p5b" in resourcegroup="rg-candidate-4-22-8rxkwjmg9fdn" to finish creating: GET https://rp.j6907648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a7ca75c5-62a9-48f1-8ad3-c65e9f548e4a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a7ca75c5-62a9-48f1-8ad3-c65e9f548e4a",
      "name": "a7ca75c5-62a9-48f1-8ad3-c65e9f548e4a",
      "status": "Failed",
      "startTime": "2026-09-18T03:50:46.402664583Z",
      "endTime": "2026-09-18T04:19:53.994096677Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 20m55s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-8rxkwjmg9fdn/cluster-candidate-4-22-8z2p5b should provision
Unexpected error:
    <*fmt.wrapError | 0xc0017682a0>: 
    failed to create HCP cluster cluster-candidate-4-22-8z2p5b: failed waiting for cluster="cluster-candidate-4-22-8z2p5b" in resourcegroup="rg-candidate-4-22-8rxkwjmg9fdn" to finish creating: GET https://rp.j6907648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a7ca75c5-62a9-48f1-8ad3-c65e9f548e4a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a7ca75c5-62a9-48f1-8ad3-c65e9f548e4a",
      "name": "a7ca75c5-62a9-48f1-8ad3-c65e9f548e4a",
      "status": "Failed",
      "startTime": "2026-09-18T03:50:46.402664583Z",
      "endTime": "2026-09-18T04:19:53.994096677Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 20m55s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--lv2mjp29lzrs/cluster-hs-pre-xxp2zq should provision
Unexpected error:
    <*fmt.wrapError | 0xc000aef7c0>: 
    failed to create HCP cluster cluster-hs-pre-xxp2zq: failed waiting for cluster="cluster-hs-pre-xxp2zq" in resourcegroup="rg-hs-pre--lv2mjp29lzrs" to finish creating: GET https://rp.j6907648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/da216ffc-9136-436b-b01d-92158b449e7c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/da216ffc-9136-436b-b01d-92158b449e7c",
      "name": "da216ffc-9136-436b-b01d-92158b449e7c",
      "status": "Failed",
      "startTime": "2026-09-18T03:50:26.310458979Z",
      "endTime": "2026-09-18T04:19:33.969303507Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m45s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--lv2mjp29lzrs/cluster-hs-pre-xxp2zq should provision
Unexpected error:
    <*fmt.wrapError | 0xc000aef7c0>: 
    failed to create HCP cluster cluster-hs-pre-xxp2zq: failed waiting for cluster="cluster-hs-pre-xxp2zq" in resourcegroup="rg-hs-pre--lv2mjp29lzrs" to finish creating: GET https://rp.j6907648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/da216ffc-9136-436b-b01d-92158b449e7c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/da216ffc-9136-436b-b01d-92158b449e7c",
      "name": "da216ffc-9136-436b-b01d-92158b449e7c",
      "status": "Failed",
      "startTime": "2026-09-18T03:50:26.310458979Z",
      "endTime": "2026-09-18T04:19:33.969303507Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 22m45s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-wj65mtt7sqkj/cluster-candidate-4-21-pk5bmj should provision
Unexpected error:
    <*fmt.wrapError | 0xc00181e300>: 
    failed to create HCP cluster cluster-candidate-4-21-pk5bmj: failed waiting for cluster="cluster-candidate-4-21-pk5bmj" in resourcegroup="rg-candidate-4-21-wj65mtt7sqkj" to finish creating: GET https://rp.j6907648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/62f044a5-f392-4d7e-8a95-8aab4d322466
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/62f044a5-f392-4d7e-8a95-8aab4d322466",
      "name": "62f044a5-f392-4d7e-8a95-8aab4d322466",
      "status": "Failed",
      "startTime": "2026-09-18T03:50:46.884252443Z",
      "endTime": "2026-09-18T04:19:53.826763956Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m57s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-wj65mtt7sqkj/cluster-candidate-4-21-pk5bmj should provision
Unexpected error:
    <*fmt.wrapError | 0xc00181e300>: 
    failed to create HCP cluster cluster-candidate-4-21-pk5bmj: failed waiting for cluster="cluster-candidate-4-21-pk5bmj" in resourcegroup="rg-candidate-4-21-wj65mtt7sqkj" to finish creating: GET https://rp.j6907648.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/62f044a5-f392-4d7e-8a95-8aab4d322466
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/62f044a5-f392-4d7e-8a95-8aab4d322466",
      "name": "62f044a5-f392-4d7e-8a95-8aab4d322466",
      "status": "Failed",
      "startTime": "2026-09-18T03:50:46.884252443Z",
      "endTime": "2026-09-18T04:19:53.826763956Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 20m57s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T04:02:41Z
  Ended:       2026-09-18T04:32:42Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6907648-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st6e0gt56ef7jnfg7trhv4ir2638kjr-n2s7g5a5a5z9k1l", pod="router-7d99594d96-6l44r", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st6e0gt56ef7jnfg7trhv4ir2638kjr-n2s7g5a5a5z9k1l/router-7d99594d96-6l44r has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T04:12:00Z
  Ended:       2026-09-18T04:22:02Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j6907648-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc001700260>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-hjgg7zlnkh76" to finish creating: GET https://rp.j2645248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bdfce236-4265-4d78-bac8-a4418dc005ef
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bdfce236-4265-4d78-bac8-a4418dc005ef",
      "name": "bdfce236-4265-4d78-bac8-a4418dc005ef",
      "status": "Failed",
      "startTime": "2026-09-18T03:43:00.850749816Z",
      "endTime": "2026-09-18T04:12:08.804551345Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m47s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc001700260>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-hjgg7zlnkh76" to finish creating: GET https://rp.j2645248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bdfce236-4265-4d78-bac8-a4418dc005ef
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bdfce236-4265-4d78-bac8-a4418dc005ef",
      "name": "bdfce236-4265-4d78-bac8-a4418dc005ef",
      "status": "Failed",
      "startTime": "2026-09-18T03:43:00.850749816Z",
      "endTime": "2026-09-18T04:12:08.804551345Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 18m47s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-gpnvkzww4bm8" to finish creating: GET https://rp.j2645248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c390c8d1-985a-452d-9ed6-80247abd9b5e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c390c8d1-985a-452d-9ed6-80247abd9b5e",
  "name": "c390c8d1-985a-452d-9ed6-80247abd9b5e",
  "status": "Failed",
  "startTime": "2026-09-18T03:56:03.231032121Z",
  "endTime": "2026-09-18T03:56:41.678977419Z",
  "error": {
    "code": "InternalServerError",
    "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
  }
}
--------------------------------------------------------------------------------
 failed to create NodePool np-one-node: failed waiting for nodepool="np-one-node" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-gpnvkzww4bm8" to finish creating: context canceled]
Expected
    <[]error | len:2, cap:2>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:132]: nodepool creation errors: [failed to create NodePool np-update-NODES: failed waiting for nodepool="np-update-NODES" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-gpnvkzww4bm8" to finish creating: GET https://rp.j2645248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c390c8d1-985a-452d-9ed6-80247abd9b5e
--------------------------------------------------------------------------------
RESPONSE 200: 200 OK
ERROR CODE: InternalServerError
--------------------------------------------------------------------------------
{
  "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/c390c8d1-985a-452d-9ed6-80247abd9b5e",
  "name": "c390c8d1-985a-452d-9ed6-80247abd9b5e",
  "status": "Failed",
  "startTime": "2026-09-18T03:56:03.231032121Z",
  "endTime": "2026-09-18T03:56:41.678977419Z",
  "error": {
    "code": "InternalServerError",
    "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
  }
}
--------------------------------------------------------------------------------
 failed to create NodePool np-one-node: failed waiting for nodepool="np-one-node" for cluster "np-update-nodes-hcp-cluster" in resourcegroup="nodepool-update-nodes-gpnvkzww4bm8" to finish creating: context canceled]
Expected
    <[]error | len:2, cap:2>: ...
to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:111]: failed to create node pool "np-zrxbbn" for cluster "cluster-hs-pre-zrxbbn"
Unexpected error:
    <*fmt.wrapError | 0xc001015700>: 
    failed to create NodePool np-zrxbbn: failed waiting for nodepool="np-zrxbbn" for cluster "cluster-hs-pre-zrxbbn" in resourcegroup="rg-hs-pre--gs2tq8nbktl2" to finish creating: GET https://rp.j2645248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4afee37d-b341-457c-963a-1bd1b8f2a81c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4afee37d-b341-457c-963a-1bd1b8f2a81c",
      "name": "4afee37d-b341-457c-963a-1bd1b8f2a81c",
      "status": "Failed",
      "startTime": "2026-09-18T03:55:06.959884593Z",
      "endTime": "2026-09-18T03:55:51.664849739Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:111]: failed to create node pool "np-zrxbbn" for cluster "cluster-hs-pre-zrxbbn"
Unexpected error:
    <*fmt.wrapError | 0xc001015700>: 
    failed to create NodePool np-zrxbbn: failed waiting for nodepool="np-zrxbbn" for cluster "cluster-hs-pre-zrxbbn" in resourcegroup="rg-hs-pre--gs2tq8nbktl2" to finish creating: GET https://rp.j2645248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4afee37d-b341-457c-963a-1bd1b8f2a81c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/4afee37d-b341-457c-963a-1bd1b8f2a81c",
      "name": "4afee37d-b341-457c-963a-1bd1b8f2a81c",
      "status": "Failed",
      "startTime": "2026-09-18T03:55:06.959884593Z",
      "endTime": "2026-09-18T03:55:51.664849739Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc0017ae060>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-r4g2f9rh44ss" to finish creating: GET https://rp.j2645248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/499b13a6-9803-4f67-82be-dcf4f0fa859c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/499b13a6-9803-4f67-82be-dcf4f0fa859c",
      "name": "499b13a6-9803-4f67-82be-dcf4f0fa859c",
      "status": "Failed",
      "startTime": "2026-09-18T03:54:50.355767916Z",
      "endTime": "2026-09-18T03:55:21.65277157Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc0017ae060>: 
    VerifyNodesReady failed: no nodes found in the cluster
    VerifyCiliumOperational failed: not all pods in kube-system namespace are running: failed to list pods in kube-system namespace: client rate limiter Wait returned an error: context deadline exceeded
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-r4g2f9rh44ss" to finish creating: GET https://rp.j2645248.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/499b13a6-9803-4f67-82be-dcf4f0fa859c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/499b13a6-9803-4f67-82be-dcf4f0fa859c",
      "name": "499b13a6-9803-4f67-82be-dcf4f0fa859c",
      "status": "Failed",
      "startTime": "2026-09-18T03:54:50.355767916Z",
      "endTime": "2026-09-18T03:55:21.65277157Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-9ptnlfp98gkt/cluster-candidate-4-21-2b86tp should provision
Unexpected error:
    <*fmt.wrapError | 0xc0008dac60>: 
    failed to create HCP cluster cluster-candidate-4-21-2b86tp: failed waiting for cluster="cluster-candidate-4-21-2b86tp" in resourcegroup="rg-candidate-4-21-9ptnlfp98gkt" to finish creating: GET https://rp.j2043264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/60784981-dab7-45ee-b0cb-ce3256c6fe9c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/60784981-dab7-45ee-b0cb-ce3256c6fe9c",
      "name": "60784981-dab7-45ee-b0cb-ce3256c6fe9c",
      "status": "Failed",
      "startTime": "2026-09-18T03:46:27.058951069Z",
      "endTime": "2026-09-18T04:15:29.848966802Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m56s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-9ptnlfp98gkt/cluster-candidate-4-21-2b86tp should provision
Unexpected error:
    <*fmt.wrapError | 0xc0008dac60>: 
    failed to create HCP cluster cluster-candidate-4-21-2b86tp: failed waiting for cluster="cluster-candidate-4-21-2b86tp" in resourcegroup="rg-candidate-4-21-9ptnlfp98gkt" to finish creating: GET https://rp.j2043264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/60784981-dab7-45ee-b0cb-ce3256c6fe9c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/60784981-dab7-45ee-b0cb-ce3256c6fe9c",
      "name": "60784981-dab7-45ee-b0cb-ce3256c6fe9c",
      "status": "Failed",
      "startTime": "2026-09-18T03:46:27.058951069Z",
      "endTime": "2026-09-18T04:15:29.848966802Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 21m56s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:293]: failed to create cluster role binding for external auth via VM
Unexpected error:
    <*fmt.wrapError | 0xc000d173e0>: 
    VM run command "e2e-runcommand-1789704441845701237" on VM "cidr-connectivity-test-test-vm" failed and cleanup also failed: Get "https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/locations/CanadaCentral/operations/67651689-75ab-4f6c-aa6e-1746fbbdc2ad?p=c6bb2de4-260b-49a6-86a7-56c69bdc0403&api-version=2024-03-01&t=639253012427558463&c=MIIHlDCCBnygAwIBAgIQTJW7e7XbMoQlWPa331WcnDANBgkqhkiG9w0BAQsFADA2MTQwMgYDVQQDEytDQ01FIEcxIFRMUyBSU0EgMjA0OCBTSEEyNTYgMjA0OSBXVVMyIENBIDAxMB4XDTI2MDgxMTE3MzMyMloXDTI3MDIwNjIzMzMyMlowQDE-MDwGA1UEAxM1YXN5bmNvcGVyYXRpb25zaWduaW5nY2VydGlmaWNhdGUubWFuYWdlbWVudC5henVyZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDeQHh5tmZ6ZYRlf_Q_PwPJqUN9t1MEgnet-7jMlsIvkZBhjMDVPNEFS1UtIM-3StxxWCffcghG31P1UcexhHelJCujRWSZYWfEIC7z_S8BBwxDhcc2peEIdNKD1O6UKvglkD8KRsPxRT-ZwQYY3qFNngixnx3EDhtWr1a9Kg46bdJJAIIzu3flYJzQzP3yVrFuRQn4iH75UvhgEByfjJaD75Ugw9JYkgO52nE5dVc2ddxoRMWzJKxp_V2Adp7VqTbMx2T8hO9PWqkltIPwoAXIubXFj3Z5A6Wj7hE7XmY_2cFJKJfspjiqwLyfnLqlv9pCLhIVdG31g8DAGZM7fc2lAgMBAAGjggSSMIIEjjCBnQYDVR0gBIGVMIGSMAwGCisGAQQBgjd7AQEwZgYKKwYBBAGCN3sCAjBYMFYGCCsGAQUFBwICMEoeSAAzADMAZQAwADEAOQAyADEALQA0AGQANgA0AC0ANABmADgAYwAtAGEAMAA1ADUALQA1AGIAZABhAGYAZgBkADUAZQAzADMAZDAMBgorBgEEAYI3ewMCMAwGCisGAQQBgjd7BAIwDAYDVR0TAQH_BAIwADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDgYDVR0PAQH_BAQDAgWgMB0GA1UdDgQWBBQbwB3CifQh36XI9i1lBe8503G1RTAfBgNVHSMEGDAWgBSs43L6A7Jznj2VyO-HW67dG6HtaDCCAbIGA1UdHwSCAakwggGlMGmgZ6BlhmNodHRwOi8vcHJpbWFyeS1jZG4ucGtpLmNvcmUud2luZG93cy5uZXQvd2VzdHVzMi9jcmxzL2NjbWV3ZXN0dXMycGtpL2NjbWV3ZXN0dXMyaWNhMDEvNjcvY3VycmVudC5jcmwwa6BpoGeGZWh0dHA6Ly9zZWNvbmRhcnktY2RuLnBraS5jb3JlLndpbmRvd3MubmV0L3dlc3R1czIvY3Jscy9jY21ld2VzdHVzMnBraS9jY21ld2VzdHVzMmljYTAxLzY3L2N1cnJlbnQuY3JsMFqgWKBWhlRodHRwOi8vY3JsLm1pY3Jvc29mdC5jb20vd2VzdHVzMi9jcmxzL2NjbWV3ZXN0dXMycGtpL2NjbWV3ZXN0dXMyaWNhMDEvNjcvY3VycmVudC5jcmwwb6BtoGuGaWh0dHA6Ly9jY21ld2VzdHVzMnBraS53ZXN0dXMyLnBraS5jb3JlLndpbmRvd3MubmV0L2NlcnRpZmljYXRlQXV0aG9yaXRpZXMvY2NtZXdlc3R1czJpY2EwMS82Ny9jdXJyZW50LmNybDCCAbcGCCsGAQUFBwEBBIIBqTCCAaUwbAYIKwYBBQUHMAKGYGh0dHA6Ly9wcmltYXJ5LWNkbi5wa2kuY29yZS53aW5kb3dzLm5ldC93ZXN0dXMyL2NhY2VydHMvY2NtZXdlc3R1czJwa2kvY2NtZXdlc3R1czJpY2EwMS9jZXJ0LmNlcjBuBggrBgEFBQcwAoZiaHR0cDovL3NlY29uZGFyeS1jZG4ucGtpLmNvcmUud2luZG93cy5uZXQvd2VzdHVzMi9jYWNlcnRzL2NjbWV3ZXN0dXMycGtpL2NjbWV3ZXN0dXMyaWNhMDEvY2VydC5jZXIwXQYIKwYBBQUHMAKGUWh0dHA6Ly9jcmwubWljcm9zb2Z0LmNvbS93ZXN0dXMyL2NhY2VydHMvY2NtZXdlc3R1czJwa2kvY2NtZXdlc3R1czJpY2EwMS9jZXJ0LmNlcjBmBggrBgEFBQcwAoZaaHR0cDovL2NjbWV3ZXN0dXMycGtpLndlc3R1czIucGtpLmNvcmUud2luZG93cy5uZXQvY2VydGlmaWNhdGVBdXRob3JpdGllcy9jY21ld2VzdHVzMmljYTAxMA0GCSqGSIb3DQEBCwUAA4IBAQACHqTwaTHdHWv9uHzVkq0ltJxzjLPIQHJnP6ndMrFnY-TIxQDrABl7eSPlINC2qk1CFExKLmjYM8aDst83BXP-yH0AHIvBrOGVdZvqFE-8KPpkqE97J5NNLNr1lLkz4vWpbjmJhLvYmureoMJjtcLpQyz-EfACVM90Lkmu8ueIkp5mjzxa_-cMa3KYwbFXLO8huXYcrhLhvs31Z9czJKjlPptL0ZE1-rBlMXGqTdZUE1CfPRoTjMMRZe_jEJYNKyxhL_5fT2mLxdlYJsMDeTU5qM-x3lA74zxVrUvC0ZvCQdK629aZQUbWV74Uzk1DmAtUiDhGvnsOW-rug_uLr621&s=2ED070N0ZTuw-f1HVDJ70D4TL-E7a_pmemS26JAcsDWdtZNt5piI-6t7rp4DAeUvveysVwPzvAf5ULh-g26b96AAV9pXZ4I1vo74l2qjDBTA1ho2Ia11r_3U1XAtiPoLjGufya8CYXOek1iwZu7oaFEZ-4uayFIei-jUkQWxZ9Jz8_HKfyzJZZA1qJtJWKMIu_-fMQRELNXCWOkBgzgrUj1MP39GcY4oa5lZoVmPJGhHGseHZ--ETS4Vun62lwfxl7XrPb2PN0L9g7S07Nil12FIWSbh_qVJ2DC5m-7fPlkYze7LvocmWvHPwMOLiIbvvBS9Ui4BX7syQlC25zGKXw&h=siGPFN1sMGFTEzhJalRHQiDtbRbqB8eZGXQRBJUWkyo": stream error: stream ID 25; CANCEL; received from peer (cancel error: Delete "https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-qhh4b47m8ncv/providers/Microsoft.Compute/virtualMachines/cidr-connectivity-test-test-vm/runCommands/e2e-runcommand-1789704441845701237?api-version=2024-03-01": stream error: stream ID 33; CANCEL; received from peer)
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:293]: failed to create cluster role binding for external auth via VM
Unexpected error:
    <*fmt.wrapError | 0xc000d173e0>: 
    VM run command "e2e-runcommand-1789704441845701237" on VM "cidr-connectivity-test-test-vm" failed and cleanup also failed: Get "https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/locations/CanadaCentral/operations/67651689-75ab-4f6c-aa6e-1746fbbdc2ad?p=c6bb2de4-260b-49a6-86a7-56c69bdc0403&api-version=2024-03-01&t=639253012427558463&c=MIIHlDCCBnygAwIBAgIQTJW7e7XbMoQlWPa331WcnDANBgkqhkiG9w0BAQsFADA2MTQwMgYDVQQDEytDQ01FIEcxIFRMUyBSU0EgMjA0OCBTSEEyNTYgMjA0OSBXVVMyIENBIDAxMB4XDTI2MDgxMTE3MzMyMloXDTI3MDIwNjIzMzMyMlowQDE-MDwGA1UEAxM1YXN5bmNvcGVyYXRpb25zaWduaW5nY2VydGlmaWNhdGUubWFuYWdlbWVudC5henVyZS5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDeQHh5tmZ6ZYRlf_Q_PwPJqUN9t1MEgnet-7jMlsIvkZBhjMDVPNEFS1UtIM-3StxxWCffcghG31P1UcexhHelJCujRWSZYWfEIC7z_S8BBwxDhcc2peEIdNKD1O6UKvglkD8KRsPxRT-ZwQYY3qFNngixnx3EDhtWr1a9Kg46bdJJAIIzu3flYJzQzP3yVrFuRQn4iH75UvhgEByfjJaD75Ugw9JYkgO52nE5dVc2ddxoRMWzJKxp_V2Adp7VqTbMx2T8hO9PWqkltIPwoAXIubXFj3Z5A6Wj7hE7XmY_2cFJKJfspjiqwLyfnLqlv9pCLhIVdG31g8DAGZM7fc2lAgMBAAGjggSSMIIEjjCBnQYDVR0gBIGVMIGSMAwGCisGAQQBgjd7AQEwZgYKKwYBBAGCN3sCAjBYMFYGCCsGAQUFBwICMEoeSAAzADMAZQAwADEAOQAyADEALQA0AGQANgA0AC0ANABmADgAYwAtAGEAMAA1ADUALQA1AGIAZABhAGYAZgBkADUAZQAzADMAZDAMBgorBgEEAYI3ewMCMAwGCisGAQQBgjd7BAIwDAYDVR0TAQH_BAIwADAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDgYDVR0PAQH_BAQDAgWgMB0GA1UdDgQWBBQbwB3CifQh36XI9i1lBe8503G1RTAfBgNVHSMEGDAWgBSs43L6A7Jznj2VyO-HW67dG6HtaDCCAbIGA1UdHwSCAakwggGlMGmgZ6BlhmNodHRwOi8vcHJpbWFyeS1jZG4ucGtpLmNvcmUud2luZG93cy5uZXQvd2VzdHVzMi9jcmxzL2NjbWV3ZXN0dXMycGtpL2NjbWV3ZXN0dXMyaWNhMDEvNjcvY3VycmVudC5jcmwwa6BpoGeGZWh0dHA6Ly9zZWNvbmRhcnktY2RuLnBraS5jb3JlLndpbmRvd3MubmV0L3dlc3R1czIvY3Jscy9jY21ld2VzdHVzMnBraS9jY21ld2VzdHVzMmljYTAxLzY3L2N1cnJlbnQuY3JsMFqgWKBWhlRodHRwOi8vY3JsLm1pY3Jvc29mdC5jb20vd2VzdHVzMi9jcmxzL2NjbWV3ZXN0dXMycGtpL2NjbWV3ZXN0dXMyaWNhMDEvNjcvY3VycmVudC5jcmwwb6BtoGuGaWh0dHA6Ly9jY21ld2VzdHVzMnBraS53ZXN0dXMyLnBraS5jb3JlLndpbmRvd3MubmV0L2NlcnRpZmljYXRlQXV0aG9yaXRpZXMvY2NtZXdlc3R1czJpY2EwMS82Ny9jdXJyZW50LmNybDCCAbcGCCsGAQUFBwEBBIIBqTCCAaUwbAYIKwYBBQUHMAKGYGh0dHA6Ly9wcmltYXJ5LWNkbi5wa2kuY29yZS53aW5kb3dzLm5ldC93ZXN0dXMyL2NhY2VydHMvY2NtZXdlc3R1czJwa2kvY2NtZXdlc3R1czJpY2EwMS9jZXJ0LmNlcjBuBggrBgEFBQcwAoZiaHR0cDovL3NlY29uZGFyeS1jZG4ucGtpLmNvcmUud2luZG93cy5uZXQvd2VzdHVzMi9jYWNlcnRzL2NjbWV3ZXN0dXMycGtpL2NjbWV3ZXN0dXMyaWNhMDEvY2VydC5jZXIwXQYIKwYBBQUHMAKGUWh0dHA6Ly9jcmwubWljcm9zb2Z0LmNvbS93ZXN0dXMyL2NhY2VydHMvY2NtZXdlc3R1czJwa2kvY2NtZXdlc3R1czJpY2EwMS9jZXJ0LmNlcjBmBggrBgEFBQcwAoZaaHR0cDovL2NjbWV3ZXN0dXMycGtpLndlc3R1czIucGtpLmNvcmUud2luZG93cy5uZXQvY2VydGlmaWNhdGVBdXRob3JpdGllcy9jY21ld2VzdHVzMmljYTAxMA0GCSqGSIb3DQEBCwUAA4IBAQACHqTwaTHdHWv9uHzVkq0ltJxzjLPIQHJnP6ndMrFnY-TIxQDrABl7eSPlINC2qk1CFExKLmjYM8aDst83BXP-yH0AHIvBrOGVdZvqFE-8KPpkqE97J5NNLNr1lLkz4vWpbjmJhLvYmureoMJjtcLpQyz-EfACVM90Lkmu8ueIkp5mjzxa_-cMa3KYwbFXLO8huXYcrhLhvs31Z9czJKjlPptL0ZE1-rBlMXGqTdZUE1CfPRoTjMMRZe_jEJYNKyxhL_5fT2mLxdlYJsMDeTU5qM-x3lA74zxVrUvC0ZvCQdK629aZQUbWV74Uzk1DmAtUiDhGvnsOW-rug_uLr621&s=2ED070N0ZTuw-f1HVDJ70D4TL-E7a_pmemS26JAcsDWdtZNt5piI-6t7rp4DAeUvveysVwPzvAf5ULh-g26b96AAV9pXZ4I1vo74l2qjDBTA1ho2Ia11r_3U1XAtiPoLjGufya8CYXOek1iwZu7oaFEZ-4uayFIei-jUkQWxZ9Jz8_HKfyzJZZA1qJtJWKMIu_-fMQRELNXCWOkBgzgrUj1MP39GcY4oa5lZoVmPJGhHGseHZ--ETS4Vun62lwfxl7XrPb2PN0L9g7S07Nil12FIWSbh_qVJ2DC5m-7fPlkYze7LvocmWvHPwMOLiIbvvBS9Ui4BX7syQlC25zGKXw&h=siGPFN1sMGFTEzhJalRHQiDtbRbqB8eZGXQRBJUWkyo": stream error: stream ID 25; CANCEL; received from peer (cancel error: Delete "https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-qhh4b47m8ncv/providers/Microsoft.Compute/virtualMachines/cidr-connectivity-test-test-vm/runCommands/e2e-runcommand-1789704441845701237?api-version=2024-03-01": stream error: stream ID 33; CANCEL; received from peer)
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T03:58:14Z
  Ended:       2026-09-18T04:08:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2043264-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st6c4ss9pg11n90barbef44mcqrigek-v20260901", pod="router-85fc7f57f6-7t8xr", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st6c4ss9pg11n90barbef44mcqrigek-v20260901/router-85fc7f57f6-7t8xr has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-18T04:00:16Z
  Ended:       2026-09-18T04:29:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2043264-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2st6c4u3st2ot5a3qc2a5p5fm6ceqr6n-t6k8m0v7e7f7n9m", pod="router-5fbdb9d6b6-97rbb", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2st6c4u3st2ot5a3qc2a5p5fm6ceqr6n-t6k8m0v7e7f7n9m/router-5fbdb9d6b6-97rbb has been in a non-ready state for longer than 5 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc0018c70e0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:59154->172.30.0.10:53: read: connection refused
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc0018c70e0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:59154->172.30.0.10:53: read: connection refused
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--vxdfqwnwx9d5/cluster-hs-pre-wvtqvz should provision
Unexpected error:
    <*fmt.wrapError | 0xc001020ba0>: 
    failed to create HCP cluster cluster-hs-pre-wvtqvz: failed waiting for cluster="cluster-hs-pre-wvtqvz" in resourcegroup="rg-hs-pre--vxdfqwnwx9d5" to finish creating: GET https://rp.j8628480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/28738f7e-bf1e-435b-b7c4-b3ff10938c74
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/28738f7e-bf1e-435b-b7c4-b3ff10938c74",
      "name": "28738f7e-bf1e-435b-b7c4-b3ff10938c74",
      "status": "Failed",
      "startTime": "2026-09-18T03:46:19.328892293Z",
      "endTime": "2026-09-18T04:15:24.561215096Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 15m37s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:82]: HCP cluster rg-hs-pre--vxdfqwnwx9d5/cluster-hs-pre-wvtqvz should provision
Unexpected error:
    <*fmt.wrapError | 0xc001020ba0>: 
    failed to create HCP cluster cluster-hs-pre-wvtqvz: failed waiting for cluster="cluster-hs-pre-wvtqvz" in resourcegroup="rg-hs-pre--vxdfqwnwx9d5" to finish creating: GET https://rp.j8628480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/28738f7e-bf1e-435b-b7c4-b3ff10938c74
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/28738f7e-bf1e-435b-b7c4-b3ff10938c74",
      "name": "28738f7e-bf1e-435b-b7c4-b3ff10938c74",
      "status": "Failed",
      "startTime": "2026-09-18T03:46:19.328892293Z",
      "endTime": "2026-09-18T04:15:24.561215096Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 15m37s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000f90b40>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-8r8jd5phj8ps" to finish creating: GET https://rp.j8628480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de938633-4e59-468a-9378-96c48d8ef3fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de938633-4e59-468a-9378-96c48d8ef3fd",
      "name": "de938633-4e59-468a-9378-96c48d8ef3fd",
      "status": "Failed",
      "startTime": "2026-09-18T03:48:43.242131083Z",
      "endTime": "2026-09-18T04:17:44.681797406Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 16m33s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000f90b40>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-8r8jd5phj8ps" to finish creating: GET https://rp.j8628480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de938633-4e59-468a-9378-96c48d8ef3fd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/de938633-4e59-468a-9378-96c48d8ef3fd",
      "name": "de938633-4e59-468a-9378-96c48d8ef3fd",
      "status": "Failed",
      "startTime": "2026-09-18T03:48:43.242131083Z",
      "endTime": "2026-09-18T04:17:44.681797406Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 16m33s ago"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0008963c0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-6htjbrlxsjhg" to finish creating: GET https://rp.j8628480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b82b6b34-867d-4037-91ad-ef65d0343bbd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b82b6b34-867d-4037-91ad-ef65d0343bbd",
      "name": "b82b6b34-867d-4037-91ad-ef65d0343bbd",
      "status": "Failed",
      "startTime": "2026-09-18T03:46:48.434685072Z",
      "endTime": "2026-09-18T04:15:54.641017549Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 15m9s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/hcp_backups.go:85]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0008963c0>: 
    failed to create HCP cluster pause-bkp-cluster: failed waiting for cluster="pause-bkp-cluster" in resourcegroup="pause-bkp-e2e-6htjbrlxsjhg" to finish creating: GET https://rp.j8628480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b82b6b34-867d-4037-91ad-ef65d0343bbd
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/b82b6b34-867d-4037-91ad-ef65d0343bbd",
      "name": "b82b6b34-867d-4037-91ad-ef65d0343bbd",
      "status": "Failed",
      "startTime": "2026-09-18T03:46:48.434685072Z",
      "endTime": "2026-09-18T04:15:54.641017549Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.33 is Partial (want Completed), started 15m9s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-dhdrlkcltwdt/cluster-candidate-5-0-tshnff should provision
Unexpected error:
    <*fmt.wrapError | 0xc000d46700>: 
    failed to create HCP cluster cluster-candidate-5-0-tshnff: failed waiting for cluster="cluster-candidate-5-0-tshnff" in resourcegroup="rg-candidate-5-0-dhdrlkcltwdt" to finish creating: GET https://rp.j8628480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/23603031-6375-471f-989c-9b2b7bce844f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/23603031-6375-471f-989c-9b2b7bce844f",
      "name": "23603031-6375-471f-989c-9b2b7bce844f",
      "status": "Failed",
      "startTime": "2026-09-18T03:46:30.045260058Z",
      "endTime": "2026-09-18T04:15:34.702663145Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 13m58s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-dhdrlkcltwdt/cluster-candidate-5-0-tshnff should provision
Unexpected error:
    <*fmt.wrapError | 0xc000d46700>: 
    failed to create HCP cluster cluster-candidate-5-0-tshnff: failed waiting for cluster="cluster-candidate-5-0-tshnff" in resourcegroup="rg-candidate-5-0-dhdrlkcltwdt" to finish creating: GET https://rp.j8628480.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/23603031-6375-471f-989c-9b2b7bce844f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/23603031-6375-471f-989c-9b2b7bce844f",
      "name": "23603031-6375-471f-989c-9b2b7bce844f",
      "status": "Failed",
      "startTime": "2026-09-18T03:46:30.045260058Z",
      "endTime": "2026-09-18T04:15:34.702663145Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 5.0.0-rc.2 is Partial (want Completed), started 13m58s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000dce3a0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-ql74qfp26s59" to finish creating: GET https://rp.j7111040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0964f917-6834-4736-8825-79c12267a054
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0964f917-6834-4736-8825-79c12267a054",
      "name": "0964f917-6834-4736-8825-79c12267a054",
      "status": "Failed",
      "startTime": "2026-09-18T03:48:34.744618724Z",
      "endTime": "2026-09-18T04:17:38.586077059Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 15m3s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/kms_key_rotation.go:89]: failed to create HCP cluster for KMS key rotation test
Unexpected error:
    <*fmt.wrapError | 0xc000dce3a0>: 
    failed to create HCP cluster kms-key-rotate-422: failed waiting for cluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-ql74qfp26s59" to finish creating: GET https://rp.j7111040.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0964f917-6834-4736-8825-79c12267a054
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/0964f917-6834-4736-8825-79c12267a054",
      "name": "0964f917-6834-4736-8825-79c12267a054",
      "status": "Failed",
      "startTime": "2026-09-18T03:48:34.744618724Z",
      "endTime": "2026-09-18T04:17:38.586077059Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 15m3s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-18T04:04:20Z
  Ended:       2026-09-18T04:18:20Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j5911552-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j5911552-svc).

time=2026-09-18T01:22:54.255Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 description="Step cluster\n  Kind: ARM\n  Template: templates/mgmt-cluster.bicep\n  Parameters: configurations/mgmt-cluster.tmpl.bicepparam"
time=2026-09-18T01:22:54.877Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2
time=2026-09-18T01:22:55.881Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 deployment=20d345ced78ed93f1fcc83ca672f15621e19f0c6572d4980fecd14d77cfa1a28 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9648512-mgmt-2%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F20d345ced78ed93f1fcc83ca672f15621e19f0c6572d4980fecd14d77cfa1a28
time=2026-09-18T01:24:56.385Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster stamp=2 err="stamp 2: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9648512-mgmt-2/providers/Microsoft.Resources/deployments/20d345ced78ed93f1fcc83ca672f15621e19f0c6572d4980fecd14d77cfa1a28/operationStatuses/08584119123100595444\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9648512-mgmt-2/providers/Microsoft.Resources/deployments/cluster-zhmgezzmxfkbc\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"BadRequest\\\",\\r\\n            \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j9648512-mgmt-2/providers/Microsoft.ContainerService/managedClusters/ci01-j9648512-mgmt-2\\\",\\r\\n            \\\"message\\\": \\\"{\\\\r\\\\n  \\\\\\\"code\\\\\\\": \\\\\\\"GetMSICredentialError\\\\\\\",\\\\r\\\\n  \\\\\\\"details\\\\\\\": null,\\\\r\\\\n  \\\\\\\"message\\\\\\\": \\\\\\\"Get managed identity credential failed. Details: Category: ClientError; Code: NotFound; SubCode: NotFound; Message: unexpected response from MSI data plane, length of returned certificate: 0; Dependency: MSI Credential; AKSTeam: RP\\\\\\\",\\\\r\\\\n  \\\\\\\"subcode\\\\\\\": \\\\\\\"\\\\\\\"\\\\r\\\\n}\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc000b28b70>: 
    clusterIssuer not ready: reason=ErrUpdateACMEAccount message=Failed to update ACME account:503 urn:ietf:params:acme:error:rateLimited: Service busy; retry later.
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:356]: Timed out after 300.000s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc000b28b70>: 
    clusterIssuer not ready: reason=ErrUpdateACMEAccount message=Failed to update ACME account:503 urn:ietf:params:acme:error:rateLimited: Service busy; retry later.
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc00114d9e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-zgncvtrh7b9x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc00114d9e0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-zgncvtrh7b9x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc00055c870>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-16-151028, targetMinor=5.1.0-0.nightly-multi-2026-09-15-135250) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-15-135250"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc00055c870>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-16-151028, targetMinor=5.1.0-0.nightly-multi-2026-09-15-135250) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-15-135250"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc002bb68a0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-t78vdzzqpblp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc002bb68a0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-t78vdzzqpblp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc000b80720>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-nz9bqqgzphzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc000b80720>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-nz9bqqgzphzd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc000f5e360>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-cx8sqpwt9s45/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc000f5e360>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-cx8sqpwt9s45/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc001376700>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-tzdswnvvvrrb" to finish creating: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:198]: failed to create node pool version 4.20.32 for back-level cluster
Unexpected error:
    <*fmt.wrapError | 0xc001376700>: 
    failed waiting for nodepool="np-ver-4-20-32" for cluster "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-tzdswnvvvrrb" to finish creating: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00298e7c0>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-9f9tbz46mct7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-9f9tbz46mct7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00298e7c0>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-9f9tbz46mct7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-9f9tbz46mct7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc002804f90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-grcsb5ww9xqw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc002804f90>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-grcsb5ww9xqw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc00011be60>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-tcgt6v" in resourcegroup="rg-np-skip-minor-tcgt6v-cwclsxtppk9g": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-tcgt6v-cwclsxtppk9g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-tcgt6v/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc00011be60>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-tcgt6v" in resourcegroup="rg-np-skip-minor-tcgt6v-cwclsxtppk9g": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-tcgt6v-cwclsxtppk9g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-tcgt6v/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc0013480c0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-mpcclt" resourcegroup="rg-np-dg-minor-mpcclt-kpfflj2pg95l": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-mpcclt-kpfflj2pg95l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-mpcclt/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc0013480c0>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-mpcclt" resourcegroup="rg-np-dg-minor-mpcclt-kpfflj2pg95l": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-mpcclt-kpfflj2pg95l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-mpcclt/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc00160a220>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-ddwqfs" in resourcegroup="rg-np-version-upgrade-ddwqfs-m7xcgsn6lcv6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-ddwqfs-m7xcgsn6lcv6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-ddwqfs/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc00160a220>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-ddwqfs" in resourcegroup="rg-np-version-upgrade-ddwqfs-m7xcgsn6lcv6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-ddwqfs-m7xcgsn6lcv6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-ddwqfs/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-xrmv7w" for cluster "cluster-nightly-4-20-xrmv7w"
Unexpected error:
    <*fmt.wrapError | 0xc000a98aa0>: 
    failed to create NodePool np-xrmv7w: failed starting nodepool creation "np-xrmv7w" for cluster "cluster-nightly-4-20-xrmv7w" in resourcegroup="rg-nightly-4-20-xx4pb26cwd2j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-xx4pb26cwd2j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-xrmv7w/nodePools/np-xrmv7w
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-xrmv7w" for cluster "cluster-nightly-4-20-xrmv7w"
Unexpected error:
    <*fmt.wrapError | 0xc000a98aa0>: 
    failed to create NodePool np-xrmv7w: failed starting nodepool creation "np-xrmv7w" for cluster "cluster-nightly-4-20-xrmv7w" in resourcegroup="rg-nightly-4-20-xx4pb26cwd2j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-xx4pb26cwd2j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-xrmv7w/nodePools/np-xrmv7w
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0014ced60>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-wvt6tvvldjdg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/80eaa57a-6c21-4104-9379-5f7032bfe26e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/80eaa57a-6c21-4104-9379-5f7032bfe26e",
      "name": "80eaa57a-6c21-4104-9379-5f7032bfe26e",
      "status": "Failed",
      "startTime": "2026-09-18T18:06:15.268087105Z",
      "endTime": "2026-09-18T18:35:15.027123676Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0014ced60>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-wvt6tvvldjdg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/80eaa57a-6c21-4104-9379-5f7032bfe26e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2/hcpOperationStatuses/80eaa57a-6c21-4104-9379-5f7032bfe26e",
      "name": "80eaa57a-6c21-4104-9379-5f7032bfe26e",
      "status": "Failed",
      "startTime": "2026-09-18T18:06:15.268087105Z",
      "endTime": "2026-09-18T18:35:15.027123676Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000de50e0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-bddwvtnwz8x8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1de5df02-d5bc-4ea1-8f35-f119420c215b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1de5df02-d5bc-4ea1-8f35-f119420c215b",
      "name": "1de5df02-d5bc-4ea1-8f35-f119420c215b",
      "status": "Failed",
      "startTime": "2026-09-18T19:54:45.082839017Z",
      "endTime": "2026-09-18T20:23:47.024709747Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000de50e0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-bddwvtnwz8x8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1de5df02-d5bc-4ea1-8f35-f119420c215b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1de5df02-d5bc-4ea1-8f35-f119420c215b",
      "name": "1de5df02-d5bc-4ea1-8f35-f119420c215b",
      "status": "Failed",
      "startTime": "2026-09-18T19:54:45.082839017Z",
      "endTime": "2026-09-18T20:23:47.024709747Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000a60580>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-65kllg7cv2w7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e3de0bda-9b1d-4eb0-bf9e-927086d1d405
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e3de0bda-9b1d-4eb0-bf9e-927086d1d405",
      "name": "e3de0bda-9b1d-4eb0-bf9e-927086d1d405",
      "status": "Failed",
      "startTime": "2026-09-18T18:06:07.854642874Z",
      "endTime": "2026-09-18T18:35:13.922823026Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000a60580>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-65kllg7cv2w7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e3de0bda-9b1d-4eb0-bf9e-927086d1d405
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e3de0bda-9b1d-4eb0-bf9e-927086d1d405",
      "name": "e3de0bda-9b1d-4eb0-bf9e-927086d1d405",
      "status": "Failed",
      "startTime": "2026-09-18T18:06:07.854642874Z",
      "endTime": "2026-09-18T18:35:13.922823026Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-w7lg9w" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc0017e2320>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-w7lg9w: failed waiting for cluster="cp-ystream-upgrade-5-0-w7lg9w" in resourcegroup="rg-cp-ystream-upgrade-5-0-whnjtcclp2gm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/58ae58c3-0f2b-4f24-99ef-9d3487e0c74e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/58ae58c3-0f2b-4f24-99ef-9d3487e0c74e",
      "name": "58ae58c3-0f2b-4f24-99ef-9d3487e0c74e",
      "status": "Failed",
      "startTime": "2026-09-18T19:01:40.681093019Z",
      "endTime": "2026-09-18T19:30:40.290116436Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-w7lg9w" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc0017e2320>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-w7lg9w: failed waiting for cluster="cp-ystream-upgrade-5-0-w7lg9w" in resourcegroup="rg-cp-ystream-upgrade-5-0-whnjtcclp2gm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/58ae58c3-0f2b-4f24-99ef-9d3487e0c74e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/58ae58c3-0f2b-4f24-99ef-9d3487e0c74e",
      "name": "58ae58c3-0f2b-4f24-99ef-9d3487e0c74e",
      "status": "Failed",
      "startTime": "2026-09-18T19:01:40.681093019Z",
      "endTime": "2026-09-18T19:30:40.290116436Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0010085a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-266qhn49mfp5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/724dbb1d-6cfa-4cff-8bf4-5e1e742772df
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/724dbb1d-6cfa-4cff-8bf4-5e1e742772df",
      "name": "724dbb1d-6cfa-4cff-8bf4-5e1e742772df",
      "status": "Failed",
      "startTime": "2026-09-18T18:06:21.25552087Z",
      "endTime": "2026-09-18T18:35:23.923631594Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0010085a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-266qhn49mfp5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/724dbb1d-6cfa-4cff-8bf4-5e1e742772df
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/724dbb1d-6cfa-4cff-8bf4-5e1e742772df",
      "name": "724dbb1d-6cfa-4cff-8bf4-5e1e742772df",
      "status": "Failed",
      "startTime": "2026-09-18T18:06:21.25552087Z",
      "endTime": "2026-09-18T18:35:23.923631594Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000abc600>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-nhxrjzhdkw4m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/954f356d-f569-48c9-8a16-ac8774f18b21
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/954f356d-f569-48c9-8a16-ac8774f18b21",
      "name": "954f356d-f569-48c9-8a16-ac8774f18b21",
      "status": "Failed",
      "startTime": "2026-09-18T19:54:57.113094659Z",
      "endTime": "2026-09-18T20:23:57.027116398Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000abc600>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-nhxrjzhdkw4m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/954f356d-f569-48c9-8a16-ac8774f18b21
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/954f356d-f569-48c9-8a16-ac8774f18b21",
      "name": "954f356d-f569-48c9-8a16-ac8774f18b21",
      "status": "Failed",
      "startTime": "2026-09-18T19:54:57.113094659Z",
      "endTime": "2026-09-18T20:23:57.027116398Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-n6q68g with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000be8cc0>: 
    failed to create HCP cluster np-version-upgrade-cluster-n6q68g: failed waiting for cluster="np-version-upgrade-cluster-n6q68g" in resourcegroup="rg-np-version-upgrade-n6q68g-z5wvsgjdr7l9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/47d83203-d5e7-4af0-adbd-1321b73b5d96
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/47d83203-d5e7-4af0-adbd-1321b73b5d96",
      "name": "47d83203-d5e7-4af0-adbd-1321b73b5d96",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:24.695867176Z",
      "endTime": "2026-09-18T20:24:27.02816156Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-n6q68g with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000be8cc0>: 
    failed to create HCP cluster np-version-upgrade-cluster-n6q68g: failed waiting for cluster="np-version-upgrade-cluster-n6q68g" in resourcegroup="rg-np-version-upgrade-n6q68g-z5wvsgjdr7l9" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/47d83203-d5e7-4af0-adbd-1321b73b5d96
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/47d83203-d5e7-4af0-adbd-1321b73b5d96",
      "name": "47d83203-d5e7-4af0-adbd-1321b73b5d96",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:24.695867176Z",
      "endTime": "2026-09-18T20:24:27.02816156Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0004ef160>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-v6jpgxxsfw4m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c804e049-d95a-4f1c-8a94-12089c6d4c28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c804e049-d95a-4f1c-8a94-12089c6d4c28",
      "name": "c804e049-d95a-4f1c-8a94-12089c6d4c28",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:46.580974029Z",
      "endTime": "2026-09-18T18:33:53.917669415Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0004ef160>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-v6jpgxxsfw4m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c804e049-d95a-4f1c-8a94-12089c6d4c28
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c804e049-d95a-4f1c-8a94-12089c6d4c28",
      "name": "c804e049-d95a-4f1c-8a94-12089c6d4c28",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:46.580974029Z",
      "endTime": "2026-09-18T18:33:53.917669415Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-v8gljqqvz6w4/cluster-candidate-4-22-ts6cw9 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0013bebc0>: 
    failed to create HCP cluster cluster-candidate-4-22-ts6cw9: failed waiting for cluster="cluster-candidate-4-22-ts6cw9" in resourcegroup="rg-candidate-4-22-v8gljqqvz6w4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6ecb16b4-c24f-4db6-986c-dc7d0a058ebe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6ecb16b4-c24f-4db6-986c-dc7d0a058ebe",
      "name": "6ecb16b4-c24f-4db6-986c-dc7d0a058ebe",
      "status": "Failed",
      "startTime": "2026-09-18T19:00:17.002894711Z",
      "endTime": "2026-09-18T19:29:20.288780423Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-v8gljqqvz6w4/cluster-candidate-4-22-ts6cw9 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0013bebc0>: 
    failed to create HCP cluster cluster-candidate-4-22-ts6cw9: failed waiting for cluster="cluster-candidate-4-22-ts6cw9" in resourcegroup="rg-candidate-4-22-v8gljqqvz6w4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6ecb16b4-c24f-4db6-986c-dc7d0a058ebe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6ecb16b4-c24f-4db6-986c-dc7d0a058ebe",
      "name": "6ecb16b4-c24f-4db6-986c-dc7d0a058ebe",
      "status": "Failed",
      "startTime": "2026-09-18T19:00:17.002894711Z",
      "endTime": "2026-09-18T19:29:20.288780423Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-hs7c8h" with version 4.20.37 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc001204dc0>: 
    failed to create HCP cluster cluster-zstream-4-20-hs7c8h: failed waiting for cluster="cluster-zstream-4-20-hs7c8h" in resourcegroup="rg-zstream-upgrade-4-20-4v6msf6jsvhm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fe569f4a-6146-4ebf-90e6-4fef2b13a008
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fe569f4a-6146-4ebf-90e6-4fef2b13a008",
      "name": "fe569f4a-6146-4ebf-90e6-4fef2b13a008",
      "status": "Failed",
      "startTime": "2026-09-18T19:00:45.232517978Z",
      "endTime": "2026-09-18T19:29:50.287676961Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-hs7c8h" with version 4.20.37 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc001204dc0>: 
    failed to create HCP cluster cluster-zstream-4-20-hs7c8h: failed waiting for cluster="cluster-zstream-4-20-hs7c8h" in resourcegroup="rg-zstream-upgrade-4-20-4v6msf6jsvhm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fe569f4a-6146-4ebf-90e6-4fef2b13a008
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fe569f4a-6146-4ebf-90e6-4fef2b13a008",
      "name": "fe569f4a-6146-4ebf-90e6-4fef2b13a008",
      "status": "Failed",
      "startTime": "2026-09-18T19:00:45.232517978Z",
      "endTime": "2026-09-18T19:29:50.287676961Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000ae2e60>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-92vpld7m4q55" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/043f530f-1cb5-4b43-ae85-55bc6d808229
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/043f530f-1cb5-4b43-ae85-55bc6d808229",
      "name": "043f530f-1cb5-4b43-ae85-55bc6d808229",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:24.023266499Z",
      "endTime": "2026-09-18T20:24:27.026850238Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000ae2e60>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-92vpld7m4q55" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/043f530f-1cb5-4b43-ae85-55bc6d808229
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/043f530f-1cb5-4b43-ae85-55bc6d808229",
      "name": "043f530f-1cb5-4b43-ae85-55bc6d808229",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:24.023266499Z",
      "endTime": "2026-09-18T20:24:27.026850238Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc0010a86c0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-ql4cwwsm948w" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-ql4cwwsm948w, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapErrors | 0xc0010a86c0>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-ql4cwwsm948w" to finish creating, caused by: timeout '30.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-20 in resource group rg-cluster-back-version-ql4cwwsm948w, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc000872e40>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-9pzlkssjh8rk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/32dd996c-52a3-42f2-90b1-59c3004ebd0e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/32dd996c-52a3-42f2-90b1-59c3004ebd0e",
      "name": "32dd996c-52a3-42f2-90b1-59c3004ebd0e",
      "status": "Failed",
      "startTime": "2026-09-18T18:59:49.139622647Z",
      "endTime": "2026-09-18T19:28:50.286578702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc000872e40>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-9pzlkssjh8rk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/32dd996c-52a3-42f2-90b1-59c3004ebd0e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/32dd996c-52a3-42f2-90b1-59c3004ebd0e",
      "name": "32dd996c-52a3-42f2-90b1-59c3004ebd0e",
      "status": "Failed",
      "startTime": "2026-09-18T18:59:49.139622647Z",
      "endTime": "2026-09-18T19:28:50.286578702Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000730dc0>: 
    failed to create HCP cluster ea-listht4pz6: failed waiting for cluster="ea-listht4pz6" in resourcegroup="ea-list-zfzjn95sp6z4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1e50a71b-2480-4fde-a3f4-65f1ffbe4dbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1e50a71b-2480-4fde-a3f4-65f1ffbe4dbf",
      "name": "1e50a71b-2480-4fde-a3f4-65f1ffbe4dbf",
      "status": "Failed",
      "startTime": "2026-09-18T19:02:22.470708216Z",
      "endTime": "2026-09-18T19:31:30.292183292Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000730dc0>: 
    failed to create HCP cluster ea-listht4pz6: failed waiting for cluster="ea-listht4pz6" in resourcegroup="ea-list-zfzjn95sp6z4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1e50a71b-2480-4fde-a3f4-65f1ffbe4dbf
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1e50a71b-2480-4fde-a3f4-65f1ffbe4dbf",
      "name": "1e50a71b-2480-4fde-a3f4-65f1ffbe4dbf",
      "status": "Failed",
      "startTime": "2026-09-18T19:02:22.470708216Z",
      "endTime": "2026-09-18T19:31:30.292183292Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000d0e460>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-gq5kj4t7v9lj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/33d2aa70-41e7-4e37-bdad-c413801367c0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/33d2aa70-41e7-4e37-bdad-c413801367c0",
      "name": "33d2aa70-41e7-4e37-bdad-c413801367c0",
      "status": "Failed",
      "startTime": "2026-09-18T18:05:54.936239942Z",
      "endTime": "2026-09-18T18:35:03.923586817Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000d0e460>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-gq5kj4t7v9lj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/33d2aa70-41e7-4e37-bdad-c413801367c0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/33d2aa70-41e7-4e37-bdad-c413801367c0",
      "name": "33d2aa70-41e7-4e37-bdad-c413801367c0",
      "status": "Failed",
      "startTime": "2026-09-18T18:05:54.936239942Z",
      "endTime": "2026-09-18T18:35:03.923586817Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0007e7760>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-krl8lpfgmlzn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4dc5125d-f2b4-4bbf-8e4e-933cde635436
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4dc5125d-f2b4-4bbf-8e4e-933cde635436",
      "name": "4dc5125d-f2b4-4bbf-8e4e-933cde635436",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:32.707548645Z",
      "endTime": "2026-09-18T18:33:33.918055388Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0007e7760>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-krl8lpfgmlzn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4dc5125d-f2b4-4bbf-8e4e-933cde635436
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/4dc5125d-f2b4-4bbf-8e4e-933cde635436",
      "name": "4dc5125d-f2b4-4bbf-8e4e-933cde635436",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:32.707548645Z",
      "endTime": "2026-09-18T18:33:33.918055388Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc00077bdc0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-tsw82p5n5s4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5ff71734-3c8e-483d-9262-a12ff7b3f337
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5ff71734-3c8e-483d-9262-a12ff7b3f337",
      "name": "5ff71734-3c8e-483d-9262-a12ff7b3f337",
      "status": "Failed",
      "startTime": "2026-09-18T19:02:11.953070547Z",
      "endTime": "2026-09-18T19:31:20.290227685Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc00077bdc0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-tsw82p5n5s4n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5ff71734-3c8e-483d-9262-a12ff7b3f337
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5ff71734-3c8e-483d-9262-a12ff7b3f337",
      "name": "5ff71734-3c8e-483d-9262-a12ff7b3f337",
      "status": "Failed",
      "startTime": "2026-09-18T19:02:11.953070547Z",
      "endTime": "2026-09-18T19:31:20.290227685Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-vnd6pm4rt5md/cluster-candidate-4-21-gkgzcb should provision
Unexpected error:
    <*fmt.wrapError | 0xc0013ec140>: 
    failed to create HCP cluster cluster-candidate-4-21-gkgzcb: failed waiting for cluster="cluster-candidate-4-21-gkgzcb" in resourcegroup="rg-candidate-4-21-vnd6pm4rt5md" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fefb2d3d-5d17-4ba0-9f25-ca66939b6a12
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fefb2d3d-5d17-4ba0-9f25-ca66939b6a12",
      "name": "fefb2d3d-5d17-4ba0-9f25-ca66939b6a12",
      "status": "Failed",
      "startTime": "2026-09-18T19:00:14.815434003Z",
      "endTime": "2026-09-18T19:29:20.286956387Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-vnd6pm4rt5md/cluster-candidate-4-21-gkgzcb should provision
Unexpected error:
    <*fmt.wrapError | 0xc0013ec140>: 
    failed to create HCP cluster cluster-candidate-4-21-gkgzcb: failed waiting for cluster="cluster-candidate-4-21-gkgzcb" in resourcegroup="rg-candidate-4-21-vnd6pm4rt5md" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fefb2d3d-5d17-4ba0-9f25-ca66939b6a12
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/fefb2d3d-5d17-4ba0-9f25-ca66939b6a12",
      "name": "fefb2d3d-5d17-4ba0-9f25-ca66939b6a12",
      "status": "Failed",
      "startTime": "2026-09-18T19:00:14.815434003Z",
      "endTime": "2026-09-18T19:29:20.286956387Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000d2c500>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-djhd95nkq4ln" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f1f0eeaa-4da5-4317-9297-5a813aab6f33
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f1f0eeaa-4da5-4317-9297-5a813aab6f33",
      "name": "f1f0eeaa-4da5-4317-9297-5a813aab6f33",
      "status": "Failed",
      "startTime": "2026-09-18T19:57:48.39283405Z",
      "endTime": "2026-09-18T20:26:57.037154264Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000d2c500>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed waiting for cluster="missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-djhd95nkq4ln" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f1f0eeaa-4da5-4317-9297-5a813aab6f33
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/f1f0eeaa-4da5-4317-9297-5a813aab6f33",
      "name": "f1f0eeaa-4da5-4317-9297-5a813aab6f33",
      "status": "Failed",
      "startTime": "2026-09-18T19:57:48.39283405Z",
      "endTime": "2026-09-18T20:26:57.037154264Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc001314da0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-rgw6mwjd2jdp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/09ad358c-d8b2-4b77-b6e1-79ace9bdc096
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/09ad358c-d8b2-4b77-b6e1-79ace9bdc096",
      "name": "09ad358c-d8b2-4b77-b6e1-79ace9bdc096",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:35.503440276Z",
      "endTime": "2026-09-18T18:33:43.918161334Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:96]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc001314da0>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-rgw6mwjd2jdp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/09ad358c-d8b2-4b77-b6e1-79ace9bdc096
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/09ad358c-d8b2-4b77-b6e1-79ace9bdc096",
      "name": "09ad358c-d8b2-4b77-b6e1-79ace9bdc096",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:35.503440276Z",
      "endTime": "2026-09-18T18:33:43.918161334Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0010724a0>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-fp6bsqq92kn8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/55377e80-bcab-47a5-9c3e-4350a26d1727
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/55377e80-bcab-47a5-9c3e-4350a26d1727",
      "name": "55377e80-bcab-47a5-9c3e-4350a26d1727",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:22.41327514Z",
      "endTime": "2026-09-18T18:33:23.916940698Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0010724a0>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed waiting for cluster="cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-fp6bsqq92kn8" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/55377e80-bcab-47a5-9c3e-4350a26d1727
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/55377e80-bcab-47a5-9c3e-4350a26d1727",
      "name": "55377e80-bcab-47a5-9c3e-4350a26d1727",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:22.41327514Z",
      "endTime": "2026-09-18T18:33:23.916940698Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-cjg6sk" with version 4.21.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000c6ed60>: 
    failed to create HCP cluster cluster-zstream-4-21-cjg6sk: failed waiting for cluster="cluster-zstream-4-21-cjg6sk" in resourcegroup="rg-zstream-upgrade-4-21-h8qwfrzsn2zp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e4023d65-ddb6-4faf-abf2-0efb01d18c22
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e4023d65-ddb6-4faf-abf2-0efb01d18c22",
      "name": "e4023d65-ddb6-4faf-abf2-0efb01d18c22",
      "status": "Failed",
      "startTime": "2026-09-18T19:00:38.211424873Z",
      "endTime": "2026-09-18T19:29:40.287386024Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-cjg6sk" with version 4.21.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000c6ed60>: 
    failed to create HCP cluster cluster-zstream-4-21-cjg6sk: failed waiting for cluster="cluster-zstream-4-21-cjg6sk" in resourcegroup="rg-zstream-upgrade-4-21-h8qwfrzsn2zp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e4023d65-ddb6-4faf-abf2-0efb01d18c22
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e4023d65-ddb6-4faf-abf2-0efb01d18c22",
      "name": "e4023d65-ddb6-4faf-abf2-0efb01d18c22",
      "status": "Failed",
      "startTime": "2026-09-18T19:00:38.211424873Z",
      "endTime": "2026-09-18T19:29:40.287386024Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00107a4a0>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-nsq6nncs9jx5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d18659c0-ccd6-4533-914c-00e40e166e6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d18659c0-ccd6-4533-914c-00e40e166e6f",
      "name": "d18659c0-ccd6-4533-914c-00e40e166e6f",
      "status": "Failed",
      "startTime": "2026-09-18T19:56:51.185500713Z",
      "endTime": "2026-09-18T20:25:57.031163299Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00107a4a0>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-nsq6nncs9jx5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d18659c0-ccd6-4533-914c-00e40e166e6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d18659c0-ccd6-4533-914c-00e40e166e6f",
      "name": "d18659c0-ccd6-4533-914c-00e40e166e6f",
      "status": "Failed",
      "startTime": "2026-09-18T19:56:51.185500713Z",
      "endTime": "2026-09-18T20:25:57.031163299Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc0016ca440>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-hx4h2xzxb7kg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e5c499c9-a7e3-4879-bf2c-b0c897b22258
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e5c499c9-a7e3-4879-bf2c-b0c897b22258",
      "name": "e5c499c9-a7e3-4879-bf2c-b0c897b22258",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:09.060511648Z",
      "endTime": "2026-09-18T20:24:17.026518831Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:98]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc0016ca440>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-hx4h2xzxb7kg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e5c499c9-a7e3-4879-bf2c-b0c897b22258
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/e5c499c9-a7e3-4879-bf2c-b0c897b22258",
      "name": "e5c499c9-a7e3-4879-bf2c-b0c897b22258",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:09.060511648Z",
      "endTime": "2026-09-18T20:24:17.026518831Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b7c320>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-ntmzhdz9dn8b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/304a6a3a-9f5b-4eef-9550-d0ba73b6ef78
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/304a6a3a-9f5b-4eef-9550-d0ba73b6ef78",
      "name": "304a6a3a-9f5b-4eef-9550-d0ba73b6ef78",
      "status": "Failed",
      "startTime": "2026-09-18T19:57:03.328284727Z",
      "endTime": "2026-09-18T20:26:07.032768851Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b7c320>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed waiting for cluster="shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-ntmzhdz9dn8b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/304a6a3a-9f5b-4eef-9550-d0ba73b6ef78
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/304a6a3a-9f5b-4eef-9550-d0ba73b6ef78",
      "name": "304a6a3a-9f5b-4eef-9550-d0ba73b6ef78",
      "status": "Failed",
      "startTime": "2026-09-18T19:57:03.328284727Z",
      "endTime": "2026-09-18T20:26:07.032768851Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000e42a80>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-x6ksv4qd5hh4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0d11158a-c610-4ba0-b070-6f011685df27
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0d11158a-c610-4ba0-b070-6f011685df27",
      "name": "0d11158a-c610-4ba0-b070-6f011685df27",
      "status": "Failed",
      "startTime": "2026-09-18T18:06:15.656789631Z",
      "endTime": "2026-09-18T18:35:23.922596176Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000e42a80>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-x6ksv4qd5hh4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0d11158a-c610-4ba0-b070-6f011685df27
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0d11158a-c610-4ba0-b070-6f011685df27",
      "name": "0d11158a-c610-4ba0-b070-6f011685df27",
      "status": "Failed",
      "startTime": "2026-09-18T18:06:15.656789631Z",
      "endTime": "2026-09-18T18:35:23.922596176Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000d3e4e0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-5l7s7j7vcxkh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6488d30f-ab0b-48ea-a0ec-7c581c0e717f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6488d30f-ab0b-48ea-a0ec-7c581c0e717f",
      "name": "6488d30f-ab0b-48ea-a0ec-7c581c0e717f",
      "status": "Failed",
      "startTime": "2026-09-18T19:02:09.380870167Z",
      "endTime": "2026-09-18T19:31:10.29248977Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000d3e4e0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-5l7s7j7vcxkh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6488d30f-ab0b-48ea-a0ec-7c581c0e717f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6488d30f-ab0b-48ea-a0ec-7c581c0e717f",
      "name": "6488d30f-ab0b-48ea-a0ec-7c581c0e717f",
      "status": "Failed",
      "startTime": "2026-09-18T19:02:09.380870167Z",
      "endTime": "2026-09-18T19:31:10.29248977Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ab4bc0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-9x5xbv4nzw8b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/33146f6d-4733-4ec4-b15d-3e685b83f09f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/33146f6d-4733-4ec4-b15d-3e685b83f09f",
      "name": "33146f6d-4733-4ec4-b15d-3e685b83f09f",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:03.940822361Z",
      "endTime": "2026-09-18T20:24:07.0250673Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ab4bc0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-9x5xbv4nzw8b" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/33146f6d-4733-4ec4-b15d-3e685b83f09f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/33146f6d-4733-4ec4-b15d-3e685b83f09f",
      "name": "33146f6d-4733-4ec4-b15d-3e685b83f09f",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:03.940822361Z",
      "endTime": "2026-09-18T20:24:07.0250673Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000e90f60>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-z57bbtc664tj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5f403839-210b-4649-acab-f3cebf6c199e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5f403839-210b-4649-acab-f3cebf6c199e",
      "name": "5f403839-210b-4649-acab-f3cebf6c199e",
      "status": "Failed",
      "startTime": "2026-09-18T18:59:56.042833825Z",
      "endTime": "2026-09-18T19:29:00.285060799Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000e90f60>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-z57bbtc664tj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5f403839-210b-4649-acab-f3cebf6c199e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/5f403839-210b-4649-acab-f3cebf6c199e",
      "name": "5f403839-210b-4649-acab-f3cebf6c199e",
      "status": "Failed",
      "startTime": "2026-09-18T18:59:56.042833825Z",
      "endTime": "2026-09-18T19:29:00.285060799Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc0004e17a0>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-qcdwg6ktcch6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b8db3dd6-3acf-4a8a-8882-bacb6f6fb9d0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b8db3dd6-3acf-4a8a-8882-bacb6f6fb9d0",
      "name": "b8db3dd6-3acf-4a8a-8882-bacb6f6fb9d0",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:24.536461143Z",
      "endTime": "2026-09-18T18:33:33.916870368Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc0004e17a0>: 
    failed to create HCP cluster private-ingress: failed waiting for cluster="private-ingress" in resourcegroup="private-ingress-qcdwg6ktcch6" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b8db3dd6-3acf-4a8a-8882-bacb6f6fb9d0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b8db3dd6-3acf-4a8a-8882-bacb6f6fb9d0",
      "name": "b8db3dd6-3acf-4a8a-8882-bacb6f6fb9d0",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:24.536461143Z",
      "endTime": "2026-09-18T18:33:33.916870368Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e9ab80>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-qp94zrf9dxjw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1869d89d-6245-485d-8118-d3fc6e9af9cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1869d89d-6245-485d-8118-d3fc6e9af9cc",
      "name": "1869d89d-6245-485d-8118-d3fc6e9af9cc",
      "status": "Failed",
      "startTime": "2026-09-18T19:56:31.378329777Z",
      "endTime": "2026-09-18T20:25:37.031824413Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e9ab80>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-qp94zrf9dxjw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1869d89d-6245-485d-8118-d3fc6e9af9cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1869d89d-6245-485d-8118-d3fc6e9af9cc",
      "name": "1869d89d-6245-485d-8118-d3fc6e9af9cc",
      "status": "Failed",
      "startTime": "2026-09-18T19:56:31.378329777Z",
      "endTime": "2026-09-18T20:25:37.031824413Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000fc6480>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-p4nctxb27lkf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/47247d9a-d966-4134-b361-76bb41af047f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/47247d9a-d966-4134-b361-76bb41af047f",
      "name": "47247d9a-d966-4134-b361-76bb41af047f",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:22.12537559Z",
      "endTime": "2026-09-18T18:33:23.915581975Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000fc6480>: 
    failed to create HCP cluster basic-cluster: failed waiting for cluster="basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-p4nctxb27lkf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/47247d9a-d966-4134-b361-76bb41af047f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/47247d9a-d966-4134-b361-76bb41af047f",
      "name": "47247d9a-d966-4134-b361-76bb41af047f",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:22.12537559Z",
      "endTime": "2026-09-18T18:33:23.915581975Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-njqww7" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000af10c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-njqww7: failed waiting for cluster="cp-ystream-upgrade-4-21-njqww7" in resourcegroup="rg-cp-ystream-upgrade-4-21-jpqnx2lrkvjw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b4fccd14-460f-43de-9e90-738bd6033ce0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b4fccd14-460f-43de-9e90-738bd6033ce0",
      "name": "b4fccd14-460f-43de-9e90-738bd6033ce0",
      "status": "Failed",
      "startTime": "2026-09-18T19:01:31.270212929Z",
      "endTime": "2026-09-18T19:30:40.29151616Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-njqww7" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000af10c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-njqww7: failed waiting for cluster="cp-ystream-upgrade-4-21-njqww7" in resourcegroup="rg-cp-ystream-upgrade-4-21-jpqnx2lrkvjw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b4fccd14-460f-43de-9e90-738bd6033ce0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b4fccd14-460f-43de-9e90-738bd6033ce0",
      "name": "b4fccd14-460f-43de-9e90-738bd6033ce0",
      "status": "Failed",
      "startTime": "2026-09-18T19:01:31.270212929Z",
      "endTime": "2026-09-18T19:30:40.29151616Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0009b0380>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-8dhk5572s4k2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0b13dfc6-e74b-46ab-86f5-cfed93dba41f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0b13dfc6-e74b-46ab-86f5-cfed93dba41f",
      "name": "0b13dfc6-e74b-46ab-86f5-cfed93dba41f",
      "status": "Failed",
      "startTime": "2026-09-18T18:07:16.233683149Z",
      "endTime": "2026-09-18T18:36:20.211533932Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0009b0380>: 
    failed waiting for cluster="agg-cluster" in resourcegroup="feature-aggregation-8dhk5572s4k2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0b13dfc6-e74b-46ab-86f5-cfed93dba41f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0b13dfc6-e74b-46ab-86f5-cfed93dba41f",
      "name": "0b13dfc6-e74b-46ab-86f5-cfed93dba41f",
      "status": "Failed",
      "startTime": "2026-09-18T18:07:16.233683149Z",
      "endTime": "2026-09-18T18:36:20.211533932Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b14ec0>: 
    failed to create HCP cluster np-dg-minor-xwhdp5: failed waiting for cluster="np-dg-minor-xwhdp5" in resourcegroup="rg-np-dg-minor-xwhdp5-592m9fw6ltl5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/35a3b7f4-5677-4ee3-8047-d4e91fdf91e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/35a3b7f4-5677-4ee3-8047-d4e91fdf91e0",
      "name": "35a3b7f4-5677-4ee3-8047-d4e91fdf91e0",
      "status": "Failed",
      "startTime": "2026-09-18T19:56:39.90066671Z",
      "endTime": "2026-09-18T20:25:47.031092558Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b14ec0>: 
    failed to create HCP cluster np-dg-minor-xwhdp5: failed waiting for cluster="np-dg-minor-xwhdp5" in resourcegroup="rg-np-dg-minor-xwhdp5-592m9fw6ltl5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/35a3b7f4-5677-4ee3-8047-d4e91fdf91e0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/35a3b7f4-5677-4ee3-8047-d4e91fdf91e0",
      "name": "35a3b7f4-5677-4ee3-8047-d4e91fdf91e0",
      "status": "Failed",
      "startTime": "2026-09-18T19:56:39.90066671Z",
      "endTime": "2026-09-18T20:25:47.031092558Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc0010da5a0>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-6ppzp5frfmgz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/18010207-11d3-4bad-8b26-bfe3951f9a05
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/18010207-11d3-4bad-8b26-bfe3951f9a05",
      "name": "18010207-11d3-4bad-8b26-bfe3951f9a05",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:25.031159118Z",
      "endTime": "2026-09-18T18:33:33.919612215Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc0010da5a0>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-6ppzp5frfmgz" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/18010207-11d3-4bad-8b26-bfe3951f9a05
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/18010207-11d3-4bad-8b26-bfe3951f9a05",
      "name": "18010207-11d3-4bad-8b26-bfe3951f9a05",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:25.031159118Z",
      "endTime": "2026-09-18T18:33:33.919612215Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-7ggpqb" with version 4.22.13 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc001088300>: 
    failed to create HCP cluster cluster-zstream-4-22-7ggpqb: failed waiting for cluster="cluster-zstream-4-22-7ggpqb" in resourcegroup="rg-zstream-upgrade-4-22-nfmkkc97chvl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/14449ce9-b1ea-49dc-820c-31ab5444c62f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/14449ce9-b1ea-49dc-820c-31ab5444c62f",
      "name": "14449ce9-b1ea-49dc-820c-31ab5444c62f",
      "status": "Failed",
      "startTime": "2026-09-18T19:00:37.092484844Z",
      "endTime": "2026-09-18T19:29:40.290112371Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-7ggpqb" with version 4.22.13 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc001088300>: 
    failed to create HCP cluster cluster-zstream-4-22-7ggpqb: failed waiting for cluster="cluster-zstream-4-22-7ggpqb" in resourcegroup="rg-zstream-upgrade-4-22-nfmkkc97chvl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/14449ce9-b1ea-49dc-820c-31ab5444c62f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/14449ce9-b1ea-49dc-820c-31ab5444c62f",
      "name": "14449ce9-b1ea-49dc-820c-31ab5444c62f",
      "status": "Failed",
      "startTime": "2026-09-18T19:00:37.092484844Z",
      "endTime": "2026-09-18T19:29:40.290112371Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0011faaa0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-smxqhhn278sn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ecd24d8c-9b96-49ef-9ee9-932851b8cc59
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ecd24d8c-9b96-49ef-9ee9-932851b8cc59",
      "name": "ecd24d8c-9b96-49ef-9ee9-932851b8cc59",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:22.39543422Z",
      "endTime": "2026-09-18T18:33:23.917604609Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0011faaa0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-smxqhhn278sn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ecd24d8c-9b96-49ef-9ee9-932851b8cc59
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ecd24d8c-9b96-49ef-9ee9-932851b8cc59",
      "name": "ecd24d8c-9b96-49ef-9ee9-932851b8cc59",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:22.39543422Z",
      "endTime": "2026-09-18T18:33:23.917604609Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-kg5zmv" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000dfa6c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-kg5zmv: failed waiting for cluster="cp-ystream-upgrade-4-22-kg5zmv" in resourcegroup="rg-cp-ystream-upgrade-4-22-gsppncf9n78j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6811d4f8-987c-40e9-a8fb-87314864cef2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6811d4f8-987c-40e9-a8fb-87314864cef2",
      "name": "6811d4f8-987c-40e9-a8fb-87314864cef2",
      "status": "Failed",
      "startTime": "2026-09-18T19:01:34.659094453Z",
      "endTime": "2026-09-18T19:30:40.290599044Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-kg5zmv" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000dfa6c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-kg5zmv: failed waiting for cluster="cp-ystream-upgrade-4-22-kg5zmv" in resourcegroup="rg-cp-ystream-upgrade-4-22-gsppncf9n78j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6811d4f8-987c-40e9-a8fb-87314864cef2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/6811d4f8-987c-40e9-a8fb-87314864cef2",
      "name": "6811d4f8-987c-40e9-a8fb-87314864cef2",
      "status": "Failed",
      "startTime": "2026-09-18T19:01:34.659094453Z",
      "endTime": "2026-09-18T19:30:40.290599044Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000dcf3c0>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-2v27j8w6flzw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3a82c0c6-4a42-4a86-a966-622844d30b20
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3a82c0c6-4a42-4a86-a966-622844d30b20",
      "name": "3a82c0c6-4a42-4a86-a966-622844d30b20",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:31.190318654Z",
      "endTime": "2026-09-18T18:33:33.918230691Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000dcf3c0>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-2v27j8w6flzw" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3a82c0c6-4a42-4a86-a966-622844d30b20
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/3a82c0c6-4a42-4a86-a966-622844d30b20",
      "name": "3a82c0c6-4a42-4a86-a966-622844d30b20",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:31.190318654Z",
      "endTime": "2026-09-18T18:33:33.918230691Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-jspqshnm6b8s/cluster-candidate-4-20-26p7kj should provision
Unexpected error:
    <*fmt.wrapError | 0xc000d98400>: 
    failed to create HCP cluster cluster-candidate-4-20-26p7kj: failed waiting for cluster="cluster-candidate-4-20-26p7kj" in resourcegroup="rg-candidate-4-20-jspqshnm6b8s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/670cbc64-10c9-40e8-ab17-ba6edf9a8da6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/670cbc64-10c9-40e8-ab17-ba6edf9a8da6",
      "name": "670cbc64-10c9-40e8-ab17-ba6edf9a8da6",
      "status": "Failed",
      "startTime": "2026-09-18T19:00:14.619385936Z",
      "endTime": "2026-09-18T19:29:20.288515017Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-jspqshnm6b8s/cluster-candidate-4-20-26p7kj should provision
Unexpected error:
    <*fmt.wrapError | 0xc000d98400>: 
    failed to create HCP cluster cluster-candidate-4-20-26p7kj: failed waiting for cluster="cluster-candidate-4-20-26p7kj" in resourcegroup="rg-candidate-4-20-jspqshnm6b8s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/670cbc64-10c9-40e8-ab17-ba6edf9a8da6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/670cbc64-10c9-40e8-ab17-ba6edf9a8da6",
      "name": "670cbc64-10c9-40e8-ab17-ba6edf9a8da6",
      "status": "Failed",
      "startTime": "2026-09-18T19:00:14.619385936Z",
      "endTime": "2026-09-18T19:29:20.288515017Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000b6d540>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-rc84t6g5h54p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/33766b12-eb33-46f0-b3b0-eea3719fedb2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/33766b12-eb33-46f0-b3b0-eea3719fedb2",
      "name": "33766b12-eb33-46f0-b3b0-eea3719fedb2",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:32.507928047Z",
      "endTime": "2026-09-18T18:33:33.91760498Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000b6d540>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-rc84t6g5h54p" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/33766b12-eb33-46f0-b3b0-eea3719fedb2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/33766b12-eb33-46f0-b3b0-eea3719fedb2",
      "name": "33766b12-eb33-46f0-b3b0-eea3719fedb2",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:32.507928047Z",
      "endTime": "2026-09-18T18:33:33.91760498Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-pc4nmpfmfd8n/cluster-candidate-5-0-8k9kgf should provision
Unexpected error:
    <*fmt.wrapError | 0xc0008977c0>: 
    failed to create HCP cluster cluster-candidate-5-0-8k9kgf: failed waiting for cluster="cluster-candidate-5-0-8k9kgf" in resourcegroup="rg-candidate-5-0-pc4nmpfmfd8n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b151a448-87ca-4125-a955-ff96a58ce701
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b151a448-87ca-4125-a955-ff96a58ce701",
      "name": "b151a448-87ca-4125-a955-ff96a58ce701",
      "status": "Failed",
      "startTime": "2026-09-18T19:00:04.863718426Z",
      "endTime": "2026-09-18T19:29:10.287329661Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-pc4nmpfmfd8n/cluster-candidate-5-0-8k9kgf should provision
Unexpected error:
    <*fmt.wrapError | 0xc0008977c0>: 
    failed to create HCP cluster cluster-candidate-5-0-8k9kgf: failed waiting for cluster="cluster-candidate-5-0-8k9kgf" in resourcegroup="rg-candidate-5-0-pc4nmpfmfd8n" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b151a448-87ca-4125-a955-ff96a58ce701
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/b151a448-87ca-4125-a955-ff96a58ce701",
      "name": "b151a448-87ca-4125-a955-ff96a58ce701",
      "status": "Failed",
      "startTime": "2026-09-18T19:00:04.863718426Z",
      "endTime": "2026-09-18T19:29:10.287329661Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-tpqdn5 at version 4.22.14
Unexpected error:
    <*fmt.wrapError | 0xc000d0e760>: 
    failed to create HCP cluster np-skip-minor-tpqdn5: failed waiting for cluster="np-skip-minor-tpqdn5" in resourcegroup="rg-np-skip-minor-tpqdn5-tc97l4b65b5k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/06ddea4f-de67-40c5-a480-9d00159e56c4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/06ddea4f-de67-40c5-a480-9d00159e56c4",
      "name": "06ddea4f-de67-40c5-a480-9d00159e56c4",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:52.428957278Z",
      "endTime": "2026-09-18T20:24:57.029688909Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-tpqdn5 at version 4.22.14
Unexpected error:
    <*fmt.wrapError | 0xc000d0e760>: 
    failed to create HCP cluster np-skip-minor-tpqdn5: failed waiting for cluster="np-skip-minor-tpqdn5" in resourcegroup="rg-np-skip-minor-tpqdn5-tc97l4b65b5k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/06ddea4f-de67-40c5-a480-9d00159e56c4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/06ddea4f-de67-40c5-a480-9d00159e56c4",
      "name": "06ddea4f-de67-40c5-a480-9d00159e56c4",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:52.428957278Z",
      "endTime": "2026-09-18T20:24:57.029688909Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-wlmbkw" with version 5.0.0-rc.1 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000a24ce0>: 
    failed to create HCP cluster cluster-zstream-5-0-wlmbkw: failed waiting for cluster="cluster-zstream-5-0-wlmbkw" in resourcegroup="rg-zstream-upgrade-5-0-88f465g4v8qg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8399e46e-e9e9-480e-b60a-122ad9e52531
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8399e46e-e9e9-480e-b60a-122ad9e52531",
      "name": "8399e46e-e9e9-480e-b60a-122ad9e52531",
      "status": "Failed",
      "startTime": "2026-09-18T19:00:37.645326719Z",
      "endTime": "2026-09-18T19:29:40.288068036Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-wlmbkw" with version 5.0.0-rc.1 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000a24ce0>: 
    failed to create HCP cluster cluster-zstream-5-0-wlmbkw: failed waiting for cluster="cluster-zstream-5-0-wlmbkw" in resourcegroup="rg-zstream-upgrade-5-0-88f465g4v8qg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8399e46e-e9e9-480e-b60a-122ad9e52531
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/8399e46e-e9e9-480e-b60a-122ad9e52531",
      "name": "8399e46e-e9e9-480e-b60a-122ad9e52531",
      "status": "Failed",
      "startTime": "2026-09-18T19:00:37.645326719Z",
      "endTime": "2026-09-18T19:29:40.288068036Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-phwsxm
Unexpected error:
    <*fmt.wrapError | 0xc0010dc9c0>: 
    failed to create HCP cluster np-noedge-phwsxm: failed waiting for cluster="np-noedge-phwsxm" in resourcegroup="rg-np-noedge-phwsxm-4bv8jxxfz58h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ac9b5fdb-7eb8-486a-9321-c2c553703d36
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ac9b5fdb-7eb8-486a-9321-c2c553703d36",
      "name": "ac9b5fdb-7eb8-486a-9321-c2c553703d36",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:33.085250323Z",
      "endTime": "2026-09-18T20:24:37.028572057Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-phwsxm
Unexpected error:
    <*fmt.wrapError | 0xc0010dc9c0>: 
    failed to create HCP cluster np-noedge-phwsxm: failed waiting for cluster="np-noedge-phwsxm" in resourcegroup="rg-np-noedge-phwsxm-4bv8jxxfz58h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ac9b5fdb-7eb8-486a-9321-c2c553703d36
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ac9b5fdb-7eb8-486a-9321-c2c553703d36",
      "name": "ac9b5fdb-7eb8-486a-9321-c2c553703d36",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:33.085250323Z",
      "endTime": "2026-09-18T20:24:37.028572057Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000b6e260>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-ld4s6tgxr5vg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/a000d2ab-54f0-4334-b41d-9e9451140d91
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/a000d2ab-54f0-4334-b41d-9e9451140d91",
      "name": "a000d2ab-54f0-4334-b41d-9e9451140d91",
      "status": "Failed",
      "startTime": "2026-09-18T19:16:16.855336614Z",
      "endTime": "2026-09-18T19:45:20.354805698Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000b6e260>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-ld4s6tgxr5vg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/a000d2ab-54f0-4334-b41d-9e9451140d91
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/a000d2ab-54f0-4334-b41d-9e9451140d91",
      "name": "a000d2ab-54f0-4334-b41d-9e9451140d91",
      "status": "Failed",
      "startTime": "2026-09-18T19:16:16.855336614Z",
      "endTime": "2026-09-18T19:45:20.354805698Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c425a0>: 
    failed to create HCP cluster np-downgrade-7d26bl: failed waiting for cluster="np-downgrade-7d26bl" in resourcegroup="rg-np-downgrade-7d26bl-7qmvc9mdqdmp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/02ee3c59-eff4-48f9-8e78-44b142853e94
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/02ee3c59-eff4-48f9-8e78-44b142853e94",
      "name": "02ee3c59-eff4-48f9-8e78-44b142853e94",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:45.411002197Z",
      "endTime": "2026-09-18T20:24:47.028298292Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c425a0>: 
    failed to create HCP cluster np-downgrade-7d26bl: failed waiting for cluster="np-downgrade-7d26bl" in resourcegroup="rg-np-downgrade-7d26bl-7qmvc9mdqdmp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/02ee3c59-eff4-48f9-8e78-44b142853e94
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/02ee3c59-eff4-48f9-8e78-44b142853e94",
      "name": "02ee3c59-eff4-48f9-8e78-44b142853e94",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:45.411002197Z",
      "endTime": "2026-09-18T20:24:47.028298292Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc0011ed1a0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d4795f3f-29d6-4f65-a818-0470837029c0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d4795f3f-29d6-4f65-a818-0470837029c0",
      "name": "d4795f3f-29d6-4f65-a818-0470837029c0",
      "status": "Failed",
      "startTime": "2026-09-18T18:05:57.828166643Z",
      "endTime": "2026-09-18T18:35:03.924193327Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc0011ed1a0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d4795f3f-29d6-4f65-a818-0470837029c0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/d4795f3f-29d6-4f65-a818-0470837029c0",
      "name": "d4795f3f-29d6-4f65-a818-0470837029c0",
      "status": "Failed",
      "startTime": "2026-09-18T18:05:57.828166643Z",
      "endTime": "2026-09-18T18:35:03.924193327Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-59kkxj with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000f90940>: 
    failed to create HCP cluster np-version-upgrade-cluster-59kkxj: failed waiting for cluster="np-version-upgrade-cluster-59kkxj" in resourcegroup="rg-np-version-upgrade-59kkxj-strjcq5cf28t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c6cb01a7-ddc6-483a-83c4-b3f161303a6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c6cb01a7-ddc6-483a-83c4-b3f161303a6f",
      "name": "c6cb01a7-ddc6-483a-83c4-b3f161303a6f",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:19.582015951Z",
      "endTime": "2026-09-18T20:24:27.027192644Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-59kkxj with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000f90940>: 
    failed to create HCP cluster np-version-upgrade-cluster-59kkxj: failed waiting for cluster="np-version-upgrade-cluster-59kkxj" in resourcegroup="rg-np-version-upgrade-59kkxj-strjcq5cf28t" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c6cb01a7-ddc6-483a-83c4-b3f161303a6f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/c6cb01a7-ddc6-483a-83c4-b3f161303a6f",
      "name": "c6cb01a7-ddc6-483a-83c4-b3f161303a6f",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:19.582015951Z",
      "endTime": "2026-09-18T20:24:27.027192644Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc0016acc40>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-t8z82lzs48sr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ee4e2ffd-758b-447c-95c9-acf961276e55
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ee4e2ffd-758b-447c-95c9-acf961276e55",
      "name": "ee4e2ffd-758b-447c-95c9-acf961276e55",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:32.431481456Z",
      "endTime": "2026-09-18T18:33:33.917230374Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc0016acc40>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-t8z82lzs48sr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ee4e2ffd-758b-447c-95c9-acf961276e55
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/ee4e2ffd-758b-447c-95c9-acf961276e55",
      "name": "ee4e2ffd-758b-447c-95c9-acf961276e55",
      "status": "Failed",
      "startTime": "2026-09-18T18:04:32.431481456Z",
      "endTime": "2026-09-18T18:33:33.917230374Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc0015aed60>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-vxq45g66n2fb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1aa4f569-8351-4ab6-8cbb-fa86fa721f61
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1aa4f569-8351-4ab6-8cbb-fa86fa721f61",
      "name": "1aa4f569-8351-4ab6-8cbb-fa86fa721f61",
      "status": "Failed",
      "startTime": "2026-09-18T18:59:43.251936528Z",
      "endTime": "2026-09-18T19:28:50.284453068Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc0015aed60>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-vxq45g66n2fb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1aa4f569-8351-4ab6-8cbb-fa86fa721f61
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/1aa4f569-8351-4ab6-8cbb-fa86fa721f61",
      "name": "1aa4f569-8351-4ab6-8cbb-fa86fa721f61",
      "status": "Failed",
      "startTime": "2026-09-18T18:59:43.251936528Z",
      "endTime": "2026-09-18T19:28:50.284453068Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-gs8fc2 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000bbc920>: 
    failed to create HCP cluster np-version-upgrade-cluster-gs8fc2: failed waiting for cluster="np-version-upgrade-cluster-gs8fc2" in resourcegroup="rg-np-version-upgrade-gs8fc2-flflx6nnvpbv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/602c1354-35fa-4664-9d86-220f8f644a46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/602c1354-35fa-4664-9d86-220f8f644a46",
      "name": "602c1354-35fa-4664-9d86-220f8f644a46",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:14.360447552Z",
      "endTime": "2026-09-18T20:24:17.027918855Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-gs8fc2 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000bbc920>: 
    failed to create HCP cluster np-version-upgrade-cluster-gs8fc2: failed waiting for cluster="np-version-upgrade-cluster-gs8fc2" in resourcegroup="rg-np-version-upgrade-gs8fc2-flflx6nnvpbv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/602c1354-35fa-4664-9d86-220f8f644a46
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/602c1354-35fa-4664-9d86-220f8f644a46",
      "name": "602c1354-35fa-4664-9d86-220f8f644a46",
      "status": "Failed",
      "startTime": "2026-09-18T19:55:14.360447552Z",
      "endTime": "2026-09-18T20:24:17.027918855Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster placement did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:319]: failed to create storage account diagnostic setting for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-d9mbtgrlrxw2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*exported.ResponseError | 0xc000a45890>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-d9mbtgrlrxw2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster/providers/Microsoft.Insights/diagnosticSettings/shoebox-storage-diag
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: SubscriptionNotRegistered
    --------------------------------------------------------------------------------
    {
      "code": "SubscriptionNotRegistered",
      "message": "The subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' is not registered to use microsoft.insights."
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:319]: failed to create storage account diagnostic setting for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-d9mbtgrlrxw2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*exported.ResponseError | 0xc000a45890>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-d9mbtgrlrxw2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster/providers/Microsoft.Insights/diagnosticSettings/shoebox-storage-diag
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: SubscriptionNotRegistered
    --------------------------------------------------------------------------------
    {
      "code": "SubscriptionNotRegistered",
      "message": "The subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' is not registered to use microsoft.insights."
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:319]: failed to create storage account diagnostic setting for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-986dm5j578hd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*exported.ResponseError | 0xc00117a2a0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-986dm5j578hd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster/providers/Microsoft.Insights/diagnosticSettings/shoebox-storage-diag
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: SubscriptionNotRegistered
    --------------------------------------------------------------------------------
    {
      "code": "SubscriptionNotRegistered",
      "message": "The subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' is not registered to use microsoft.insights."
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:319]: failed to create storage account diagnostic setting for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-986dm5j578hd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*exported.ResponseError | 0xc00117a2a0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-986dm5j578hd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster/providers/Microsoft.Insights/diagnosticSettings/shoebox-storage-diag
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: SubscriptionNotRegistered
    --------------------------------------------------------------------------------
    {
      "code": "SubscriptionNotRegistered",
      "message": "The subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' is not registered to use microsoft.insights."
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:319]: failed to create storage account diagnostic setting for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-92rgrk84fv8d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*exported.ResponseError | 0xc00088e570>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-92rgrk84fv8d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster/providers/Microsoft.Insights/diagnosticSettings/shoebox-storage-diag
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: SubscriptionNotRegistered
    --------------------------------------------------------------------------------
    {
      "code": "SubscriptionNotRegistered",
      "message": "The subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' is not registered to use microsoft.insights."
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:319]: failed to create storage account diagnostic setting for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-92rgrk84fv8d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*exported.ResponseError | 0xc00088e570>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-92rgrk84fv8d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster/providers/Microsoft.Insights/diagnosticSettings/shoebox-storage-diag
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: SubscriptionNotRegistered
    --------------------------------------------------------------------------------
    {
      "code": "SubscriptionNotRegistered",
      "message": "The subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' is not registered to use microsoft.insights."
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0xc002d105c0>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-wbftvl6pxn6v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f5be950-3d76-400d-bba8-80953e9b85fb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f5be950-3d76-400d-bba8-80953e9b85fb",
      "name": "9f5be950-3d76-400d-bba8-80953e9b85fb",
      "status": "Failed",
      "startTime": "2026-09-18T07:41:11.725184381Z",
      "endTime": "2026-09-18T08:00:14.656645268Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:270]: failed to create node pool oidc-wi-np
Unexpected error:
    <*fmt.wrapError | 0xc002d105c0>: 
    failed to create NodePool oidc-wi-np: failed waiting for nodepool="oidc-wi-np" for cluster "oidc-wi-cluster" in resourcegroup="oidc-wi-wbftvl6pxn6v" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f5be950-3d76-400d-bba8-80953e9b85fb
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9f5be950-3d76-400d-bba8-80953e9b85fb",
      "name": "9f5be950-3d76-400d-bba8-80953e9b85fb",
      "status": "Failed",
      "startTime": "2026-09-18T07:41:11.725184381Z",
      "endTime": "2026-09-18T08:00:14.656645268Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc001092648>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-tgqlbhrjcjzd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f4e2d962-cb10-4c5d-b5e9-88c9efab7468
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f4e2d962-cb10-4c5d-b5e9-88c9efab7468",
      "name": "f4e2d962-cb10-4c5d-b5e9-88c9efab7468",
      "status": "Failed",
      "startTime": "2026-09-18T02:15:45.572313063Z",
      "endTime": "2026-09-18T02:34:50.003063171Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:182]: failed to verify cilium is running and nodes are Ready for cluster "cilium-cl"
Unexpected error:
    <*errors.joinError | 0xc001092648>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cl" in resourcegroup="cni-cilium-tgqlbhrjcjzd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f4e2d962-cb10-4c5d-b5e9-88c9efab7468
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/f4e2d962-cb10-4c5d-b5e9-88c9efab7468",
      "name": "f4e2d962-cb10-4c5d-b5e9-88c9efab7468",
      "status": "Failed",
      "startTime": "2026-09-18T02:15:45.572313063Z",
      "endTime": "2026-09-18T02:34:50.003063171Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.32
Unexpected error:
    <*fmt.wrapError | 0xc000efdee0>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-dd2djv" in resourcegroup="rg-np-version-upgrade-dd2djv-f5pt9zl6csx7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34fcd870-f035-4d43-a07e-056005a0cd12
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34fcd870-f035-4d43-a07e-056005a0cd12",
      "name": "34fcd870-f035-4d43-a07e-056005a0cd12",
      "status": "Failed",
      "startTime": "2026-09-18T03:23:36.136458591Z",
      "endTime": "2026-09-18T03:42:38.777220989Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-21 with version 4.21.32
Unexpected error:
    <*fmt.wrapError | 0xc000efdee0>: 
    failed to create NodePool npupgrade-4-21: failed waiting for nodepool="npupgrade-4-21" for cluster "np-version-upgrade-cluster-dd2djv" in resourcegroup="rg-np-version-upgrade-dd2djv-f5pt9zl6csx7" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34fcd870-f035-4d43-a07e-056005a0cd12
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/34fcd870-f035-4d43-a07e-056005a0cd12",
      "name": "34fcd870-f035-4d43-a07e-056005a0cd12",
      "status": "Failed",
      "startTime": "2026-09-18T03:23:36.136458591Z",
      "endTime": "2026-09-18T03:42:38.777220989Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:319]: failed to create storage account diagnostic setting for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-sxsc7bxhvj2n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*exported.ResponseError | 0xc0012e0990>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-sxsc7bxhvj2n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster/providers/Microsoft.Insights/diagnosticSettings/shoebox-storage-diag
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: SubscriptionNotRegistered
    --------------------------------------------------------------------------------
    {
      "code": "SubscriptionNotRegistered",
      "message": "The subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' is not registered to use microsoft.insights."
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:319]: failed to create storage account diagnostic setting for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-sxsc7bxhvj2n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*exported.ResponseError | 0xc0012e0990>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-sxsc7bxhvj2n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster/providers/Microsoft.Insights/diagnosticSettings/shoebox-storage-diag
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: SubscriptionNotRegistered
    --------------------------------------------------------------------------------
    {
      "code": "SubscriptionNotRegistered",
      "message": "The subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' is not registered to use microsoft.insights."
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:738]: failed to create nodepool npdg-422-420
Unexpected error:
    <*fmt.wrapError | 0xc000d0cd20>: 
    failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-hg2tcz" in resourcegroup="rg-np-dg-minor-hg2tcz-n7pnlmn2w4pv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bd7e5bf4-841e-4289-9a4b-a34d164ba80f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bd7e5bf4-841e-4289-9a4b-a34d164ba80f",
      "name": "bd7e5bf4-841e-4289-9a4b-a34d164ba80f",
      "status": "Failed",
      "startTime": "2026-09-18T03:32:13.01856929Z",
      "endTime": "2026-09-18T03:51:16.292258995Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:738]: failed to create nodepool npdg-422-420
Unexpected error:
    <*fmt.wrapError | 0xc000d0cd20>: 
    failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-hg2tcz" in resourcegroup="rg-np-dg-minor-hg2tcz-n7pnlmn2w4pv" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bd7e5bf4-841e-4289-9a4b-a34d164ba80f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/bd7e5bf4-841e-4289-9a4b-a34d164ba80f",
      "name": "bd7e5bf4-841e-4289-9a4b-a34d164ba80f",
      "status": "Failed",
      "startTime": "2026-09-18T03:32:13.01856929Z",
      "endTime": "2026-09-18T03:51:16.292258995Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000860220>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-rm9vjp5gmczk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df01aa2b-a5c5-4899-b645-5ce58f5f8cee
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df01aa2b-a5c5-4899-b645-5ce58f5f8cee",
      "name": "df01aa2b-a5c5-4899-b645-5ce58f5f8cee",
      "status": "Failed",
      "startTime": "2026-09-18T02:08:07.344249043Z",
      "endTime": "2026-09-18T02:37:10.01628355Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000860220>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-rm9vjp5gmczk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df01aa2b-a5c5-4899-b645-5ce58f5f8cee
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/df01aa2b-a5c5-4899-b645-5ce58f5f8cee",
      "name": "df01aa2b-a5c5-4899-b645-5ce58f5f8cee",
      "status": "Failed",
      "startTime": "2026-09-18T02:08:07.344249043Z",
      "endTime": "2026-09-18T02:37:10.01628355Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:319]: failed to create storage account diagnostic setting for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-cxcwb4twxbz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*exported.ResponseError | 0xc001176ff0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-cxcwb4twxbz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster/providers/Microsoft.Insights/diagnosticSettings/shoebox-storage-diag
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: SubscriptionNotRegistered
    --------------------------------------------------------------------------------
    {
      "code": "SubscriptionNotRegistered",
      "message": "The subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' is not registered to use microsoft.insights."
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:319]: failed to create storage account diagnostic setting for cluster /subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-cxcwb4twxbz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
Unexpected error:
    <*exported.ResponseError | 0xc001176ff0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-cxcwb4twxbz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster/providers/Microsoft.Insights/diagnosticSettings/shoebox-storage-diag
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: SubscriptionNotRegistered
    --------------------------------------------------------------------------------
    {
      "code": "SubscriptionNotRegistered",
      "message": "The subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' is not registered to use microsoft.insights."
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-pd9r72" for cluster "cluster-candidate-4-20-pd9r72"
Unexpected error:
    <*fmt.wrapError | 0xc001005b20>: 
    failed to create NodePool np-pd9r72: failed waiting for nodepool="np-pd9r72" for cluster "cluster-candidate-4-20-pd9r72" in resourcegroup="rg-candidate-4-20-7nvxhnmpb6qr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e62533b3-e87f-4306-b7fa-7e8f2f79508a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e62533b3-e87f-4306-b7fa-7e8f2f79508a",
      "name": "e62533b3-e87f-4306-b7fa-7e8f2f79508a",
      "status": "Failed",
      "startTime": "2026-09-18T02:55:35.193527856Z",
      "endTime": "2026-09-18T03:14:42.13127378Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-pd9r72" for cluster "cluster-candidate-4-20-pd9r72"
Unexpected error:
    <*fmt.wrapError | 0xc001005b20>: 
    failed to create NodePool np-pd9r72: failed waiting for nodepool="np-pd9r72" for cluster "cluster-candidate-4-20-pd9r72" in resourcegroup="rg-candidate-4-20-7nvxhnmpb6qr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e62533b3-e87f-4306-b7fa-7e8f2f79508a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/e62533b3-e87f-4306-b7fa-7e8f2f79508a",
      "name": "e62533b3-e87f-4306-b7fa-7e8f2f79508a",
      "status": "Failed",
      "startTime": "2026-09-18T02:55:35.193527856Z",
      "endTime": "2026-09-18T03:14:42.13127378Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-23-6cpntrp4snvc/cluster-nightly-4-23-t8pvmw should provision
Unexpected error:
    <*fmt.wrapError | 0xc00039c500>: 
    failed to create HCP cluster cluster-nightly-4-23-t8pvmw: failed starting cluster creation "cluster-nightly-4-23-t8pvmw" in resourcegroup="rg-nightly-4-23-6cpntrp4snvc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-6cpntrp4snvc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-23-t8pvmw
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-23-6cpntrp4snvc/cluster-nightly-4-23-t8pvmw should provision
Unexpected error:
    <*fmt.wrapError | 0xc00039c500>: 
    failed to create HCP cluster cluster-nightly-4-23-t8pvmw: failed starting cluster creation "cluster-nightly-4-23-t8pvmw" in resourcegroup="rg-nightly-4-23-6cpntrp4snvc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-6cpntrp4snvc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-23-t8pvmw
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc0008ac220>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-gp6ml4ltbxq9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-gp6ml4ltbxq9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc0008ac220>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-gp6ml4ltbxq9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-gp6ml4ltbxq9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000806300>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-m52zk49cfnnp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-m52zk49cfnnp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000806300>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-m52zk49cfnnp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-m52zk49cfnnp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00019ecc0>: 
    failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-fh8hvw7ft644": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-fh8hvw7ft644/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00019ecc0>: 
    failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-fh8hvw7ft644": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-fh8hvw7ft644/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000f6e100>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-g2znk7bl28vh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g2znk7bl28vh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000f6e100>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-g2znk7bl28vh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-g2znk7bl28vh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000a102e0>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-5t8m268g2ktr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-5t8m268g2ktr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000a102e0>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-5t8m268g2ktr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-5t8m268g2ktr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000b6a6a0>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-qk8wdqf7vwt6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-qk8wdqf7vwt6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000b6a6a0>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-qk8wdqf7vwt6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-qk8wdqf7vwt6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc000e89ce0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc000e89ce0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-scjr8k" at install version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc00108e260>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-scjr8k: failed starting cluster creation "cp-ystream-upgrade-4-23-scjr8k" in resourcegroup="rg-cp-ystream-upgrade-4-23-pp97jbcxvkl7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-pp97jbcxvkl7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-scjr8k
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-scjr8k" at install version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc00108e260>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-scjr8k: failed starting cluster creation "cp-ystream-upgrade-4-23-scjr8k" in resourcegroup="rg-cp-ystream-upgrade-4-23-pp97jbcxvkl7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-pp97jbcxvkl7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-scjr8k
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-q7f6s4" at install version 4.21.0-0.nightly-multi-2026-09-17-004700
Unexpected error:
    <*fmt.wrapError | 0xc000d122a0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-q7f6s4: failed starting cluster creation "cp-ystream-upgrade-4-22-q7f6s4" in resourcegroup="rg-cp-ystream-upgrade-4-22-8bwnk6dktglb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-8bwnk6dktglb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-q7f6s4
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-q7f6s4" at install version 4.21.0-0.nightly-multi-2026-09-17-004700
Unexpected error:
    <*fmt.wrapError | 0xc000d122a0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-q7f6s4: failed starting cluster creation "cp-ystream-upgrade-4-22-q7f6s4" in resourcegroup="rg-cp-ystream-upgrade-4-22-8bwnk6dktglb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-8bwnk6dktglb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-q7f6s4
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-0-5rvr88pqswlm/cluster-nightly-5-0-8rxs7j should provision
Unexpected error:
    <*fmt.wrapError | 0xc000722280>: 
    failed to create HCP cluster cluster-nightly-5-0-8rxs7j: failed starting cluster creation "cluster-nightly-5-0-8rxs7j" in resourcegroup="rg-nightly-5-0-5rvr88pqswlm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-5rvr88pqswlm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-0-8rxs7j
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-0-5rvr88pqswlm/cluster-nightly-5-0-8rxs7j should provision
Unexpected error:
    <*fmt.wrapError | 0xc000722280>: 
    failed to create HCP cluster cluster-nightly-5-0-8rxs7j: failed starting cluster creation "cluster-nightly-5-0-8rxs7j" in resourcegroup="rg-nightly-5-0-5rvr88pqswlm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-5rvr88pqswlm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-0-8rxs7j
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc0008e20e0>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-jksvh7q9n6qs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-jksvh7q9n6qs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc0008e20e0>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-jksvh7q9n6qs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-jksvh7q9n6qs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-21-6jldpkvl8srb/cluster-nightly-4-21-lcl4f8 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001500100>: 
    failed to create HCP cluster cluster-nightly-4-21-lcl4f8: failed starting cluster creation "cluster-nightly-4-21-lcl4f8" in resourcegroup="rg-nightly-4-21-6jldpkvl8srb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-6jldpkvl8srb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-21-lcl4f8
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-21-6jldpkvl8srb/cluster-nightly-4-21-lcl4f8 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001500100>: 
    failed to create HCP cluster cluster-nightly-4-21-lcl4f8: failed starting cluster creation "cluster-nightly-4-21-lcl4f8" in resourcegroup="rg-nightly-4-21-6jldpkvl8srb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-6jldpkvl8srb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-21-lcl4f8
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000332240>: 
    failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-zsvgzkhfdgg2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-zsvgzkhfdgg2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000332240>: 
    failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-zsvgzkhfdgg2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-zsvgzkhfdgg2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-1-dtsnl8rgrzrg/cluster-nightly-5-1-47qlh5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0005c05c0>: 
    failed to create HCP cluster cluster-nightly-5-1-47qlh5: failed starting cluster creation "cluster-nightly-5-1-47qlh5" in resourcegroup="rg-nightly-5-1-dtsnl8rgrzrg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-dtsnl8rgrzrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-1-47qlh5
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-1-dtsnl8rgrzrg/cluster-nightly-5-1-47qlh5 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0005c05c0>: 
    failed to create HCP cluster cluster-nightly-5-1-47qlh5: failed starting cluster creation "cluster-nightly-5-1-47qlh5" in resourcegroup="rg-nightly-5-1-dtsnl8rgrzrg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-dtsnl8rgrzrg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-1-47qlh5
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc001018260>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-q87tv2dcjtrs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-q87tv2dcjtrs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc001018260>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-q87tv2dcjtrs": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-q87tv2dcjtrs/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:105]: failed to begin creating HCP cluster "admin-cred-lifecycle-wvjvqc"
Unexpected error:
    <*fmt.wrapError | 0xc000f78220>: 
    failed starting cluster creation "admin-cred-lifecycle-wvjvqc" in resourcegroup="admin-credential-lifecycle-test-hcvns8c6tffl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hcvns8c6tffl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-wvjvqc
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:105]: failed to begin creating HCP cluster "admin-cred-lifecycle-wvjvqc"
Unexpected error:
    <*fmt.wrapError | 0xc000f78220>: 
    failed starting cluster creation "admin-cred-lifecycle-wvjvqc" in resourcegroup="admin-credential-lifecycle-test-hcvns8c6tffl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hcvns8c6tffl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-wvjvqc
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000e80120>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-87cdm8lltpfw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-87cdm8lltpfw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000e80120>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-87cdm8lltpfw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-87cdm8lltpfw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc00118a260>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-ln4schhnc59t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-ln4schhnc59t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc00118a260>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-ln4schhnc59t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-ln4schhnc59t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000a7e460>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-jh9dxccjnhlt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-jh9dxccjnhlt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000a7e460>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-jh9dxccjnhlt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-jh9dxccjnhlt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000660600>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-tm274vd7kh49": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-tm274vd7kh49/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000660600>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-tm274vd7kh49": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-tm274vd7kh49/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0010100e0>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-qzw9gg97gq44": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-qzw9gg97gq44/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0010100e0>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-qzw9gg97gq44": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-qzw9gg97gq44/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000cdc940>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-lfp5nqml9vq2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-lfp5nqml9vq2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000cdc940>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-lfp5nqml9vq2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-lfp5nqml9vq2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-2zfwx7" at install version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc000788740>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-2zfwx7: failed starting cluster creation "cp-ystream-upgrade-4-21-2zfwx7" in resourcegroup="rg-cp-ystream-upgrade-4-21-chltnnssx9l9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-chltnnssx9l9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-2zfwx7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-2zfwx7" at install version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc000788740>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-2zfwx7: failed starting cluster creation "cp-ystream-upgrade-4-21-2zfwx7" in resourcegroup="rg-cp-ystream-upgrade-4-21-chltnnssx9l9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-chltnnssx9l9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-2zfwx7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc001284320>: 
    failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-7nfk9kblbq5v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-7nfk9kblbq5v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc001284320>: 
    failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-7nfk9kblbq5v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-7nfk9kblbq5v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-ct45g8" at install version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc000e08400>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-ct45g8: failed starting cluster creation "cp-ystream-upgrade-5-0-ct45g8" in resourcegroup="rg-cp-ystream-upgrade-5-0-8wgrfjrhz827": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-8wgrfjrhz827/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-ct45g8
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-ct45g8" at install version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc000e08400>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-ct45g8: failed starting cluster creation "cp-ystream-upgrade-5-0-ct45g8" in resourcegroup="rg-cp-ystream-upgrade-5-0-8wgrfjrhz827": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-8wgrfjrhz827/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-ct45g8
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d18100>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-9p8lhkrrwxlr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-9p8lhkrrwxlr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d18100>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-9p8lhkrrwxlr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-9p8lhkrrwxlr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-22-q9dmmpg8slh8/cluster-nightly-4-22-cwmzl7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c102c0>: 
    failed to create HCP cluster cluster-nightly-4-22-cwmzl7: failed starting cluster creation "cluster-nightly-4-22-cwmzl7" in resourcegroup="rg-nightly-4-22-q9dmmpg8slh8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-q9dmmpg8slh8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-22-cwmzl7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-22-q9dmmpg8slh8/cluster-nightly-4-22-cwmzl7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c102c0>: 
    failed to create HCP cluster cluster-nightly-4-22-cwmzl7: failed starting cluster creation "cluster-nightly-4-22-cwmzl7" in resourcegroup="rg-nightly-4-22-q9dmmpg8slh8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-q9dmmpg8slh8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-22-cwmzl7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00088c5a0>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-cgng7lgtlrhk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-cgng7lgtlrhk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00088c5a0>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-cgng7lgtlrhk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-cgng7lgtlrhk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000992e20>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-7psrwg2xhdxk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-7psrwg2xhdxk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000992e20>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-7psrwg2xhdxk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-7psrwg2xhdxk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000a06a20>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-dpsml89bt4jr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-dpsml89bt4jr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000a06a20>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-dpsml89bt4jr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-dpsml89bt4jr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc000984180>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-lc76mwwz2m4r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-lc76mwwz2m4r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc000984180>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-lc76mwwz2m4r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-lc76mwwz2m4r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-s8bnbc" at install version 5.0.0-0.nightly-multi-2026-09-16-151028
Unexpected error:
    <*fmt.wrapError | 0xc0008a81e0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-s8bnbc: failed starting cluster creation "cp-ystream-upgrade-5-1-s8bnbc" in resourcegroup="rg-cp-ystream-upgrade-5-1-gt6bnlnww2lj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-gt6bnlnww2lj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-s8bnbc
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-s8bnbc" at install version 5.0.0-0.nightly-multi-2026-09-16-151028
Unexpected error:
    <*fmt.wrapError | 0xc0008a81e0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-s8bnbc: failed starting cluster creation "cp-ystream-upgrade-5-1-s8bnbc" in resourcegroup="rg-cp-ystream-upgrade-5-1-gt6bnlnww2lj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-gt6bnlnww2lj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-s8bnbc
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc001458160>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-462w74wptjnv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-462w74wptjnv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc001458160>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-462w74wptjnv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-462w74wptjnv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-hh4kqh at version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc000e303e0>: 
    failed to create HCP cluster np-skip-minor-hh4kqh: failed starting cluster creation "np-skip-minor-hh4kqh" in resourcegroup="rg-np-skip-minor-hh4kqh-g8ngpxdpgdxv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-hh4kqh-g8ngpxdpgdxv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-hh4kqh
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-hh4kqh at version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc000e303e0>: 
    failed to create HCP cluster np-skip-minor-hh4kqh: failed starting cluster creation "np-skip-minor-hh4kqh" in resourcegroup="rg-np-skip-minor-hh4kqh-g8ngpxdpgdxv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-hh4kqh-g8ngpxdpgdxv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-hh4kqh
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc000e86cc0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-v5tthrvp4l2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc000e86cc0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-v5tthrvp4l2l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc0011fc200>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-cm746plzmjgp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-cm746plzmjgp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc0011fc200>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-cm746plzmjgp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-cm746plzmjgp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000f9e0c0>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-lszv6zzms7rw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-lszv6zzms7rw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000f9e0c0>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-lszv6zzms7rw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-lszv6zzms7rw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0010100e0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-qwjk9ftzxk8b": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qwjk9ftzxk8b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0010100e0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-qwjk9ftzxk8b": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-qwjk9ftzxk8b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-q825bz with version 4.21.0-0.nightly-multi-2026-09-17-004700
Unexpected error:
    <*fmt.wrapError | 0xc000a251e0>: 
    failed to create HCP cluster np-version-upgrade-cluster-q825bz: failed starting cluster creation "np-version-upgrade-cluster-q825bz" in resourcegroup="rg-np-version-upgrade-q825bz-x27zct4fpv9n": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-q825bz-x27zct4fpv9n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-q825bz
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-q825bz with version 4.21.0-0.nightly-multi-2026-09-17-004700
Unexpected error:
    <*fmt.wrapError | 0xc000a251e0>: 
    failed to create HCP cluster np-version-upgrade-cluster-q825bz: failed starting cluster creation "np-version-upgrade-cluster-q825bz" in resourcegroup="rg-np-version-upgrade-q825bz-x27zct4fpv9n": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-q825bz-x27zct4fpv9n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-q825bz
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000804820>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-hfrqt4znr8ww": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-hfrqt4znr8ww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000804820>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-hfrqt4znr8ww": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-hfrqt4znr8ww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-20-stz6qmm9c9pm/cluster-nightly-4-20-jhmr4f should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b7c160>: 
    failed to create HCP cluster cluster-nightly-4-20-jhmr4f: failed starting cluster creation "cluster-nightly-4-20-jhmr4f" in resourcegroup="rg-nightly-4-20-stz6qmm9c9pm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-stz6qmm9c9pm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-jhmr4f
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-20-stz6qmm9c9pm/cluster-nightly-4-20-jhmr4f should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b7c160>: 
    failed to create HCP cluster cluster-nightly-4-20-jhmr4f: failed starting cluster creation "cluster-nightly-4-20-jhmr4f" in resourcegroup="rg-nightly-4-20-stz6qmm9c9pm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-stz6qmm9c9pm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-jhmr4f
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000bba1c0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-xm5czf8ql9ll": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-xm5czf8ql9ll/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000bba1c0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-xm5czf8ql9ll": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-xm5czf8ql9ll/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0010fe300>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-68hbxkjb8pb5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-68hbxkjb8pb5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0010fe300>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-68hbxkjb8pb5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-68hbxkjb8pb5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009a29c0>: 
    failed to create HCP cluster np-dg-minor-88hvpm: failed starting cluster creation "np-dg-minor-88hvpm" in resourcegroup="rg-np-dg-minor-88hvpm-hhs8h4j7xp4c": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-88hvpm-hhs8h4j7xp4c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-88hvpm
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009a29c0>: 
    failed to create HCP cluster np-dg-minor-88hvpm: failed starting cluster creation "np-dg-minor-88hvpm" in resourcegroup="rg-np-dg-minor-88hvpm-hhs8h4j7xp4c": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-88hvpm-hhs8h4j7xp4c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-88hvpm
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000fb4180>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-qbtxlbn5dcbl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-qbtxlbn5dcbl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000fb4180>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-qbtxlbn5dcbl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-qbtxlbn5dcbl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000fd0100>: 
    failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-5sqvdl7975w5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-5sqvdl7975w5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000fd0100>: 
    failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-5sqvdl7975w5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-5sqvdl7975w5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000a80140>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-hndxmlgn7dsk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-hndxmlgn7dsk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000a80140>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-hndxmlgn7dsk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-hndxmlgn7dsk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc00156e300>: 
    failed to create HCP cluster ea-listvrnw2h: failed starting cluster creation "ea-listvrnw2h" in resourcegroup="ea-list-gqc5zfmw9m5t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-gqc5zfmw9m5t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listvrnw2h
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc00156e300>: 
    failed to create HCP cluster ea-listvrnw2h: failed starting cluster creation "ea-listvrnw2h" in resourcegroup="ea-list-gqc5zfmw9m5t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-gqc5zfmw9m5t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listvrnw2h
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000ad80e0>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-x4gcgll7nktk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-x4gcgll7nktk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000ad80e0>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-x4gcgll7nktk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-x4gcgll7nktk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000333e80>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-mkvhgjv66g9w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-mkvhgjv66g9w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000333e80>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-mkvhgjv66g9w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-mkvhgjv66g9w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc0004badc0>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-78mvmfhw2cmf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-78mvmfhw2cmf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc0004badc0>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-78mvmfhw2cmf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-78mvmfhw2cmf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.33
Unexpected error:
    <*fmt.wrapErrors | 0xc002292870>: 
    failed waiting for nodepool="npupgrade-4-21" in cluster="np-version-upgrade-cluster-nl8qnf" resourcegroup="rg-np-version-upgrade-nl8qnf-jssxs5wlfz7w" to finish updating, caused by: timeout '45.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool npupgrade-4-21 in cluster np-version-upgrade-cluster-nl8qnf in resource group rg-np-version-upgrade-nl8qnf-jssxs5wlfz7w, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:195]: failed to upgrade node pool npupgrade-4-21 to version 4.21.33
Unexpected error:
    <*fmt.wrapErrors | 0xc002292870>: 
    failed waiting for nodepool="npupgrade-4-21" in cluster="np-version-upgrade-cluster-nl8qnf" resourcegroup="rg-np-version-upgrade-nl8qnf-jssxs5wlfz7w" to finish updating, caused by: timeout '45.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool npupgrade-4-21 in cluster np-version-upgrade-cluster-nl8qnf in resource group rg-np-version-upgrade-nl8qnf-jssxs5wlfz7w, error: context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-h29zrk" for cluster "cluster-candidate-4-21-h29zrk"
Unexpected error:
    <*fmt.wrapError | 0xc0009df3c0>: 
    failed to create NodePool np-h29zrk: failed waiting for nodepool="np-h29zrk" for cluster "cluster-candidate-4-21-h29zrk" in resourcegroup="rg-candidate-4-21-k7gt4s25kh5h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eba02645-cd7e-4330-9e90-bb7a800d75d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eba02645-cd7e-4330-9e90-bb7a800d75d5",
      "name": "eba02645-cd7e-4330-9e90-bb7a800d75d5",
      "status": "Failed",
      "startTime": "2026-09-18T02:38:03.543368376Z",
      "endTime": "2026-09-18T02:57:09.03735448Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-h29zrk" for cluster "cluster-candidate-4-21-h29zrk"
Unexpected error:
    <*fmt.wrapError | 0xc0009df3c0>: 
    failed to create NodePool np-h29zrk: failed waiting for nodepool="np-h29zrk" for cluster "cluster-candidate-4-21-h29zrk" in resourcegroup="rg-candidate-4-21-k7gt4s25kh5h" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eba02645-cd7e-4330-9e90-bb7a800d75d5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/eba02645-cd7e-4330-9e90-bb7a800d75d5",
      "name": "eba02645-cd7e-4330-9e90-bb7a800d75d5",
      "status": "Failed",
      "startTime": "2026-09-18T02:38:03.543368376Z",
      "endTime": "2026-09-18T02:57:09.03735448Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred