fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc001038980>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-bjdvsjn552jk" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-00.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e74a3a8c-8b66-4ef6-aeeb-66083699b0c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e74a3a8c-8b66-4ef6-aeeb-66083699b0c5",
      "name": "e74a3a8c-8b66-4ef6-aeeb-66083699b0c5",
      "status": "Failed",
      "startTime": "2026-09-24T00:25:27.786214979Z",
      "endTime": "2026-09-24T00:36:16.168471473Z",
      "error": {
        "code": "InvalidResource",
        "message": "DataPlaneIdentitiesPermissionsValidation: MissingRequiredPermissions: Data plane operators missing required permissions: [{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-bjdvsjn552jk/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-20\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-00-16/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]},{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-bjdvsjn552jk/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-20/subnets/customer-vnet-subnet-4-20\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-00-16/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/read\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"},{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]}]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc001038980>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-bjdvsjn552jk" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-00.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e74a3a8c-8b66-4ef6-aeeb-66083699b0c5
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/e74a3a8c-8b66-4ef6-aeeb-66083699b0c5",
      "name": "e74a3a8c-8b66-4ef6-aeeb-66083699b0c5",
      "status": "Failed",
      "startTime": "2026-09-24T00:25:27.786214979Z",
      "endTime": "2026-09-24T00:36:16.168471473Z",
      "error": {
        "code": "InvalidResource",
        "message": "DataPlaneIdentitiesPermissionsValidation: MissingRequiredPermissions: Data plane operators missing required permissions: [{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-bjdvsjn552jk/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-20\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-00-16/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]},{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-bjdvsjn552jk/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-20/subnets/customer-vnet-subnet-4-20\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard3-00-16/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/read\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"},{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]}]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:357]: Timed out after 300.003s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc001cc06c0>: 
    clusterIssuer not ready: reason=ErrUpdateACMEAccount message=Failed to update ACME account:503 urn:ietf:params:acme:error:rateLimited: Service busy; retry later.
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:357]: Timed out after 300.003s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc001cc06c0>: 
    clusterIssuer not ready: reason=ErrUpdateACMEAccount message=Failed to update ACME account:503 urn:ietf:params:acme:error:rateLimited: Service busy; retry later.
    ...

time=2026-09-23T23:25:15.442Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-23T23:25:17.399Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-23T23:25:18.602Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=72a5eef9024c0fb803ba8efe57568ddcaa2d06a84e61744f2eeaeba881052bfe portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j9983616%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F72a5eef9024c0fb803ba8efe57568ddcaa2d06a84e61744f2eeaeba881052bfe
time=2026-09-23T23:27:49.191Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j9983616/providers/Microsoft.Resources/deployments/72a5eef9024c0fb803ba8efe57568ddcaa2d06a84e61744f2eeaeba881052bfe/operationStatuses/08584114009676534307\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"code\\\": \\\"BadRequest\\\",\\r\\n  \\\"message\\\": \\\"The zone 'aro-hcp-dev-shard1-slot-01.hcp.osadev.cloud' is not available.\\\"\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc000a9f0c0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-94qwsncfglvx" to finish creating: GET https://rp.aro-hcp-dev-shard1-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9a2c4ba6-7b02-45b6-b246-27301c3c7b44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9a2c4ba6-7b02-45b6-b246-27301c3c7b44",
      "name": "9a2c4ba6-7b02-45b6-b246-27301c3c7b44",
      "status": "Failed",
      "startTime": "2026-09-23T23:07:02.752849186Z",
      "endTime": "2026-09-23T23:26:11.541839688Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:109]: failed to create ARM64 nodepool "arm64-vm-np-1" with VM size "Standard_D2plds_v6"
Unexpected error:
    <*fmt.wrapError | 0xc000a9f0c0>: 
    failed to create NodePool arm64-vm-np-1: failed waiting for nodepool="arm64-vm-np-1" for cluster "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-94qwsncfglvx" to finish creating: GET https://rp.aro-hcp-dev-shard1-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9a2c4ba6-7b02-45b6-b246-27301c3c7b44
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/9a2c4ba6-7b02-45b6-b246-27301c3c7b44",
      "name": "9a2c4ba6-7b02-45b6-b246-27301c3c7b44",
      "status": "Failed",
      "startTime": "2026-09-23T23:07:02.752849186Z",
      "endTime": "2026-09-23T23:26:11.541839688Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T23:16:20Z
  Ended:       2026-09-23T23:41:14Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j7507200-svc", component="backend", name="datadump", region="centralus", severity="warning"
  Description: Backend controller workqueue datadump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-24T00:08:12Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j7507200-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-rpfwd", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-rpfwd (namespace arobit) on cluster ci01-j7507200-svc has averaged 126.1% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-96r455f6cgr8/cluster-candidate-4-21-tc5xmb should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ba4aa0>: 
    failed to create HCP cluster cluster-candidate-4-21-tc5xmb: failed waiting for cluster="cluster-candidate-4-21-tc5xmb" in resourcegroup="rg-candidate-4-21-96r455f6cgr8" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c2add970-69a8-4470-839c-381250f38990
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c2add970-69a8-4470-839c-381250f38990",
      "name": "c2add970-69a8-4470-839c-381250f38990",
      "status": "Failed",
      "startTime": "2026-09-23T22:56:17.620493086Z",
      "endTime": "2026-09-23T23:25:22.742938899Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-96r455f6cgr8/cluster-candidate-4-21-tc5xmb should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ba4aa0>: 
    failed to create HCP cluster cluster-candidate-4-21-tc5xmb: failed waiting for cluster="cluster-candidate-4-21-tc5xmb" in resourcegroup="rg-candidate-4-21-96r455f6cgr8" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c2add970-69a8-4470-839c-381250f38990
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/c2add970-69a8-4470-839c-381250f38990",
      "name": "c2add970-69a8-4470-839c-381250f38990",
      "status": "Failed",
      "startTime": "2026-09-23T22:56:17.620493086Z",
      "endTime": "2026-09-23T23:25:22.742938899Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001362d00>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-fvfnk745g57p" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/277377a5-4429-4698-a365-74f878b05216
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/277377a5-4429-4698-a365-74f878b05216",
      "name": "277377a5-4429-4698-a365-74f878b05216",
      "status": "Failed",
      "startTime": "2026-09-23T23:00:00.50151725Z",
      "endTime": "2026-09-23T23:29:02.685864461Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001362d00>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-fvfnk745g57p" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/277377a5-4429-4698-a365-74f878b05216
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/277377a5-4429-4698-a365-74f878b05216",
      "name": "277377a5-4429-4698-a365-74f878b05216",
      "status": "Failed",
      "startTime": "2026-09-23T23:00:00.50151725Z",
      "endTime": "2026-09-23T23:29:02.685864461Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-q7fxr6l2s4gv/cluster-candidate-4-23-8fnk5c should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f92780>: 
    failed to create HCP cluster cluster-candidate-4-23-8fnk5c: failed waiting for cluster="cluster-candidate-4-23-8fnk5c" in resourcegroup="rg-candidate-4-23-q7fxr6l2s4gv" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f137ea92-b35a-4536-ad97-053b360f7eb9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f137ea92-b35a-4536-ad97-053b360f7eb9",
      "name": "f137ea92-b35a-4536-ad97-053b360f7eb9",
      "status": "Failed",
      "startTime": "2026-09-23T22:55:37.049841827Z",
      "endTime": "2026-09-23T23:24:43.011652216Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-q7fxr6l2s4gv/cluster-candidate-4-23-8fnk5c should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f92780>: 
    failed to create HCP cluster cluster-candidate-4-23-8fnk5c: failed waiting for cluster="cluster-candidate-4-23-8fnk5c" in resourcegroup="rg-candidate-4-23-q7fxr6l2s4gv" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f137ea92-b35a-4536-ad97-053b360f7eb9
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/f137ea92-b35a-4536-ad97-053b360f7eb9",
      "name": "f137ea92-b35a-4536-ad97-053b360f7eb9",
      "status": "Failed",
      "startTime": "2026-09-23T22:55:37.049841827Z",
      "endTime": "2026-09-23T23:24:43.011652216Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000696ea0>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-cbz2zvj9pgcc" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8aecc921-f63e-4605-92ac-6c0825ca5e73
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8aecc921-f63e-4605-92ac-6c0825ca5e73",
      "name": "8aecc921-f63e-4605-92ac-6c0825ca5e73",
      "status": "Failed",
      "startTime": "2026-09-23T23:22:17.979672195Z",
      "endTime": "2026-09-23T23:51:22.771798672Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000696ea0>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-cbz2zvj9pgcc" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8aecc921-f63e-4605-92ac-6c0825ca5e73
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/8aecc921-f63e-4605-92ac-6c0825ca5e73",
      "name": "8aecc921-f63e-4605-92ac-6c0825ca5e73",
      "status": "Failed",
      "startTime": "2026-09-23T23:22:17.979672195Z",
      "endTime": "2026-09-23T23:51:22.771798672Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--8w2v2h7tbr7x/cluster-hs-pre-9brzc9 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000854de0>: 
    failed to create HCP cluster cluster-hs-pre-9brzc9: failed waiting for cluster="cluster-hs-pre-9brzc9" in resourcegroup="rg-hs-pre--8w2v2h7tbr7x" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6abdd772-bc76-4739-9454-989e477a880a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6abdd772-bc76-4739-9454-989e477a880a",
      "name": "6abdd772-bc76-4739-9454-989e477a880a",
      "status": "Failed",
      "startTime": "2026-09-23T22:56:17.609215386Z",
      "endTime": "2026-09-23T23:25:22.953425865Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--8w2v2h7tbr7x/cluster-hs-pre-9brzc9 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000854de0>: 
    failed to create HCP cluster cluster-hs-pre-9brzc9: failed waiting for cluster="cluster-hs-pre-9brzc9" in resourcegroup="rg-hs-pre--8w2v2h7tbr7x" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6abdd772-bc76-4739-9454-989e477a880a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6abdd772-bc76-4739-9454-989e477a880a",
      "name": "6abdd772-bc76-4739-9454-989e477a880a",
      "status": "Failed",
      "startTime": "2026-09-23T22:56:17.609215386Z",
      "endTime": "2026-09-23T23:25:22.953425865Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001006fc0>: 
    failed to create HCP cluster np-dg-minor-4p9zqn: failed waiting for cluster="np-dg-minor-4p9zqn" in resourcegroup="rg-np-dg-minor-4p9zqn-9pzvmjjxxgwg" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95b275e8-f395-482d-b8d4-ca393de42355
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95b275e8-f395-482d-b8d4-ca393de42355",
      "name": "95b275e8-f395-482d-b8d4-ca393de42355",
      "status": "Failed",
      "startTime": "2026-09-23T23:20:11.073191713Z",
      "endTime": "2026-09-23T23:49:12.786080022Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001006fc0>: 
    failed to create HCP cluster np-dg-minor-4p9zqn: failed waiting for cluster="np-dg-minor-4p9zqn" in resourcegroup="rg-np-dg-minor-4p9zqn-9pzvmjjxxgwg" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95b275e8-f395-482d-b8d4-ca393de42355
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/95b275e8-f395-482d-b8d4-ca393de42355",
      "name": "95b275e8-f395-482d-b8d4-ca393de42355",
      "status": "Failed",
      "startTime": "2026-09-23T23:20:11.073191713Z",
      "endTime": "2026-09-23T23:49:12.786080022Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000464ae0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-8l6vvszb7492" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/53f6605f-6b79-4454-8e28-fd239f8916df
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/53f6605f-6b79-4454-8e28-fd239f8916df",
      "name": "53f6605f-6b79-4454-8e28-fd239f8916df",
      "status": "Failed",
      "startTime": "2026-09-23T22:58:01.487898788Z",
      "endTime": "2026-09-23T23:27:02.7845863Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000464ae0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-8l6vvszb7492" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/53f6605f-6b79-4454-8e28-fd239f8916df
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/53f6605f-6b79-4454-8e28-fd239f8916df",
      "name": "53f6605f-6b79-4454-8e28-fd239f8916df",
      "status": "Failed",
      "startTime": "2026-09-23T22:58:01.487898788Z",
      "endTime": "2026-09-23T23:27:02.7845863Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc0009068a0>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-fx77xd4jlzgc" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/62671277-0174-4102-8a19-fd81304df39c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/62671277-0174-4102-8a19-fd81304df39c",
      "name": "62671277-0174-4102-8a19-fd81304df39c",
      "status": "Failed",
      "startTime": "2026-09-23T22:56:18.488865764Z",
      "endTime": "2026-09-23T23:25:22.804222005Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc0009068a0>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-fx77xd4jlzgc" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/62671277-0174-4102-8a19-fd81304df39c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/62671277-0174-4102-8a19-fd81304df39c",
      "name": "62671277-0174-4102-8a19-fd81304df39c",
      "status": "Failed",
      "startTime": "2026-09-23T22:56:18.488865764Z",
      "endTime": "2026-09-23T23:25:22.804222005Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000bf1200>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-dx59g4bq6jrp" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a97c4cb5-238d-43ff-b943-3f685b71b6f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a97c4cb5-238d-43ff-b943-3f685b71b6f1",
      "name": "a97c4cb5-238d-43ff-b943-3f685b71b6f1",
      "status": "Failed",
      "startTime": "2026-09-23T22:56:16.987949919Z",
      "endTime": "2026-09-23T23:25:22.924061821Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nodepool_active_versions.go:89]: failed to create HCP cluster "v20261001" via v20261001preview
Unexpected error:
    <*fmt.wrapError | 0xc000bf1200>: 
    failed to create HCP cluster v20261001: failed waiting for cluster="v20261001" in resourcegroup="v20261001preview-dx59g4bq6jrp" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a97c4cb5-238d-43ff-b943-3f685b71b6f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/a97c4cb5-238d-43ff-b943-3f685b71b6f1",
      "name": "a97c4cb5-238d-43ff-b943-3f685b71b6f1",
      "status": "Failed",
      "startTime": "2026-09-23T22:56:16.987949919Z",
      "endTime": "2026-09-23T23:25:22.924061821Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc001372520>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-ff8h8wnjvp8f" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2410d929-9d91-4c03-aaf0-1103e0c4f386
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2410d929-9d91-4c03-aaf0-1103e0c4f386",
      "name": "2410d929-9d91-4c03-aaf0-1103e0c4f386",
      "status": "Failed",
      "startTime": "2026-09-23T22:56:17.324238915Z",
      "endTime": "2026-09-23T23:25:22.950789091Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc001372520>: 
    failed to create HCP cluster cilium-cl: failed waiting for cluster="cilium-cl" in resourcegroup="cni-cilium-ff8h8wnjvp8f" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2410d929-9d91-4c03-aaf0-1103e0c4f386
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/2410d929-9d91-4c03-aaf0-1103e0c4f386",
      "name": "2410d929-9d91-4c03-aaf0-1103e0c4f386",
      "status": "Failed",
      "startTime": "2026-09-23T22:56:17.324238915Z",
      "endTime": "2026-09-23T23:25:22.950789091Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0015ce480>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-xcpdnghpn9qj" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4981ec5a-b923-4f46-a09c-4542a63d5e8e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4981ec5a-b923-4f46-a09c-4542a63d5e8e",
      "name": "4981ec5a-b923-4f46-a09c-4542a63d5e8e",
      "status": "Failed",
      "startTime": "2026-09-23T23:22:18.709940835Z",
      "endTime": "2026-09-23T23:51:22.784957829Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0015ce480>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-xcpdnghpn9qj" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4981ec5a-b923-4f46-a09c-4542a63d5e8e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/4981ec5a-b923-4f46-a09c-4542a63d5e8e",
      "name": "4981ec5a-b923-4f46-a09c-4542a63d5e8e",
      "status": "Failed",
      "startTime": "2026-09-23T23:22:18.709940835Z",
      "endTime": "2026-09-23T23:51:22.784957829Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T23:23:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6515968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.157:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6515968/providers/microsoft.monitor/accounts/services-j6515968", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7955558448-96rhz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-23T23:23:17Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6515968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.157:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6515968/providers/microsoft.monitor/accounts/services-j6515968", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7955558448-96rhz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-23T23:31:17Z
  Ended:       2026-09-23T23:38:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6515968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.157:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6515968/providers/microsoft.monitor/accounts/services-j6515968", namespace="klusterlet-2t10muqh402iu8cpp36j8dibe4fqfcp3", pod="arohcp-monitor-kube-state-metrics-7955558448-96rhz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t10muqh402iu8cpp36j8dibe4fqfcp3/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-23T23:32:18Z
  Ended:       2026-09-23T23:38:18Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j6515968-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.157:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6515968/providers/microsoft.monitor/accounts/services-j6515968", namespace="klusterlet-2t10nsc5krmo0ki0tnfsecad8tsn2pgc", pod="arohcp-monitor-kube-state-metrics-7955558448-96rhz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t10nsc5krmo0ki0tnfsecad8tsn2pgc/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T22:58:18Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j6515968-mgmt-2", component="kubernetes-infrastructure", namespace="open-cluster-management-agent-addon", pod="klusterlet-addon-workmgr-749558849-pbxrh", region="westus3", severity="warning"
  Description: Pod open-cluster-management-agent-addon/klusterlet-addon-workmgr-749558849-pbxrh has been in a non-ready state for longer than 5 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T23:32:20Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j6515968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.157:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6515968/providers/microsoft.monitor/accounts/services-j6515968", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7955558448-96rhz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-23T23:33:16Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j6515968-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.157:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j6515968/providers/microsoft.monitor/accounts/services-j6515968", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7955558448-96rhz", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.

time=2026-09-23T22:31:07.038Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-23T22:31:08.814Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-23T22:31:10.117Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=cd29b4bde2b5b7602cb211d98c814688cd52f3aab95b368210e0a8cfb85babb1 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8206336%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fcd29b4bde2b5b7602cb211d98c814688cd52f3aab95b368210e0a8cfb85babb1
time=2026-09-23T22:34:11.270Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8206336/providers/Microsoft.Resources/deployments/cd29b4bde2b5b7602cb211d98c814688cd52f3aab95b368210e0a8cfb85babb1/operationStatuses/08584114042161978286\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"code\\\": \\\"BadRequest\\\",\\r\\n  \\\"message\\\": \\\"The zone 'aro-hcp-dev-shard1-slot-01.hcp.osadev.cloud' is not available.\\\"\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T22:48:48Z
  Ended:       2026-09-23T23:00:45Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j1922944-mgmt-1", component="service-memory", container="kube-applier", namespace="kube-applier", pod="kube-applier-869c9489f8-dbnnd", severity="warning", team="hcp-sl"
  Description: Container kube-applier in pod kube-applier-869c9489f8-dbnnd (namespace kube-applier) on cluster ci01-j1922944-mgmt-1 has averaged 124.9% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-23T23:16:44Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j1922944-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-z6smt", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-z6smt (namespace arobit) on cluster ci01-j1922944-svc has averaged 124.8% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T23:11:40Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j8888448-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-m4cdw", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-m4cdw (namespace arobit) on cluster ci01-j8888448-svc has averaged 124.2% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

time=2026-09-23T21:07:35.163Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-23T21:07:36.837Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-23T21:07:38.387Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=5d60488de7a83700a82e6fe6e3ffcdc28baad600277f95b8571fc3d6f39546ee portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j4221952%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F5d60488de7a83700a82e6fe6e3ffcdc28baad600277f95b8571fc3d6f39546ee
time=2026-09-23T21:10:39.591Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j4221952/providers/Microsoft.Resources/deployments/5d60488de7a83700a82e6fe6e3ffcdc28baad600277f95b8571fc3d6f39546ee/operationStatuses/08584114092280138717\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"code\\\": \\\"BadRequest\\\",\\r\\n  \\\"message\\\": \\\"The zone 'aro-hcp-dev-shard1-slot-01.hcp.osadev.cloud' is not available.\\\"\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T22:47:02Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j8872448-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-z6lp4", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-z6lp4 (namespace arobit) on cluster ci01-j8872448-svc has averaged 125.7% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T21:52:45Z
  Ended:       2026-09-23T22:01:44Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j4706560-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j4706560-svc).

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T22:30:23Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j4706560-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-rsvxn", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-rsvxn (namespace arobit) on cluster ci01-j4706560-svc has averaged 123.6% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

time=2026-09-23T20:41:08.730Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra description="Step infra\n  Kind: ARM\n  Template: templates/region.bicep\n  Parameters: configurations/region.tmpl.bicepparam"
time=2026-09-23T20:41:10.239Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra
time=2026-09-23T20:41:11.440Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra deployment=c45a9cb8ea1f8b2c4d688227f5292bcfda7025c58f2a3979dc36c7e654c8c701 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j8827264%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2Fc45a9cb8ea1f8b2c4d688227f5292bcfda7025c58f2a3979dc36c7e654c8c701
time=2026-09-23T20:43:42.398Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Region resourceGroup=regional step=infra err="failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8827264/providers/Microsoft.Resources/deployments/c45a9cb8ea1f8b2c4d688227f5292bcfda7025c58f2a3979dc36c7e654c8c701/operationStatuses/08584114108148792156\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"BadRequest\",\n        \"message\": \"{\\r\\n  \\\"code\\\": \\\"BadRequest\\\",\\r\\n  \\\"message\\\": \\\"The zone 'aro-hcp-dev-shard1-slot-01.hcp.osadev.cloud' is not available.\\\"\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-ldnr6s"
Unexpected error:
    <*fmt.wrapError | 0xc00084af40>: 
    DNS for route host agnhost-e2e-sample-app-k5xxw.apps.aro.v1c9v0y2y6w8p7n.b97m.aro-hcp-dev-shard0-slot-01.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-k5xxw.apps.aro.v1c9v0y2y6w8p7n.b97m.aro-hcp-dev-shard0-slot-01.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-k5xxw.apps.aro.v1c9v0y2y6w8p7n.b97m.aro-hcp-dev-shard0-slot-01.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-ldnr6s"
Unexpected error:
    <*fmt.wrapError | 0xc00084af40>: 
    DNS for route host agnhost-e2e-sample-app-k5xxw.apps.aro.v1c9v0y2y6w8p7n.b97m.aro-hcp-dev-shard0-slot-01.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-k5xxw.apps.aro.v1c9v0y2y6w8p7n.b97m.aro-hcp-dev-shard0-slot-01.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-k5xxw.apps.aro.v1c9v0y2y6w8p7n.b97m.aro-hcp-dev-shard0-slot-01.hcp.osadev.cloud on 172.30.0.10:53: dial udp 172.30.0.10:53: i/o timeout): context deadline exceeded
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0004d5460>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-8fx9vqqhjrvz" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ca16a72-f387-4f26-a7d9-a173356a7d9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ca16a72-f387-4f26-a7d9-a173356a7d9b",
      "name": "5ca16a72-f387-4f26-a7d9-a173356a7d9b",
      "status": "Failed",
      "startTime": "2026-09-23T20:52:19.857894326Z",
      "endTime": "2026-09-23T21:11:21.209971005Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:103]: failed to create node pool "np-1" for fips-enabled cluster "fips-enabled-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0004d5460>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "fips-enabled-cluster" in resourcegroup="fips-enabled-8fx9vqqhjrvz" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ca16a72-f387-4f26-a7d9-a173356a7d9b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5ca16a72-f387-4f26-a7d9-a173356a7d9b",
      "name": "5ca16a72-f387-4f26-a7d9-a173356a7d9b",
      "status": "Failed",
      "startTime": "2026-09-23T20:52:19.857894326Z",
      "endTime": "2026-09-23T21:11:21.209971005Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:78]: failed to create customer resources for cilium CNI cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ae83c0>: 
    failed to create customer-infra: failed creating deployment "customer-infra-cilium-cl-kvs4jp" in resourcegroup="cni-cilium-vbnz9zxk9mkz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/cni-cilium-vbnz9zxk9mkz/providers/Microsoft.Resources/deployments/customer-infra-cilium-cl-kvs4jp
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: DeploymentActive
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentActive",
        "message": "The deployment with resource id '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-vbnz9zxk9mkz/providers/Microsoft.Resources/deployments/customer-infra-cilium-cl-kvs4jp' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '9/23/2026 8:41:53 PM' with correlationId '5bc6c0ba-196c-4242-b0a0-9410a2c0ee3b', and will expire at '9/30/2026 8:41:53 PM' if it does not complete before then. Please see https://aka.ms/arm-deploy-resources for usage details."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:78]: failed to create customer resources for cilium CNI cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ae83c0>: 
    failed to create customer-infra: failed creating deployment "customer-infra-cilium-cl-kvs4jp" in resourcegroup="cni-cilium-vbnz9zxk9mkz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/cni-cilium-vbnz9zxk9mkz/providers/Microsoft.Resources/deployments/customer-infra-cilium-cl-kvs4jp
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: DeploymentActive
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "DeploymentActive",
        "message": "The deployment with resource id '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-vbnz9zxk9mkz/providers/Microsoft.Resources/deployments/customer-infra-cilium-cl-kvs4jp' cannot be saved, because this would overwrite an existing deployment which is still active. If this is an intentional re-deployment, please wait for the original deployment to complete, or cancel it before re-attempting the operation. The previous deployment was started at '9/23/2026 8:41:53 PM' with correlationId '5bc6c0ba-196c-4242-b0a0-9410a2c0ee3b', and will expire at '9/30/2026 8:41:53 PM' if it does not complete before then. Please see https://aka.ms/arm-deploy-resources for usage details."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 600.002s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:157 with:
RunVMCommand failed for ingress connectivity test
Unexpected error:
    <*errors.errorString | 0xc002901d30>: 
    run command "e2e-runcommand-1790197778134945960" failed with exit code 6 on VM "private-ingress-test-vm" (output: "000")
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:166]: Timed out after 600.002s.
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:157 with:
RunVMCommand failed for ingress connectivity test
Unexpected error:
    <*errors.errorString | 0xc002901d30>: 
    run command "e2e-runcommand-1790197778134945960" failed with exit code 6 on VM "private-ingress-test-vm" (output: "000")
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T21:26:02Z
  Ended:       2026-09-23T21:34:02Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j4857344-svc", component="frontend", method="post", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/requestadmincredential", severity="warning"
  Description: More than 5% of frontend requests for post /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/requestadmincredential exceeded 1 second over the past 30 minutes (cluster ci01-j4857344-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-hv5jpf95qtjp/cluster-candidate-4-21-xnt7bq should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f4a320>: 
    failed to create HCP cluster cluster-candidate-4-21-xnt7bq: failed waiting for cluster="cluster-candidate-4-21-xnt7bq" in resourcegroup="rg-candidate-4-21-hv5jpf95qtjp" to finish creating: GET https://rp.aro-hcp-dev-shard1-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4e39e4f-715f-4e75-a247-c1f9fd0ee98c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4e39e4f-715f-4e75-a247-c1f9fd0ee98c",
      "name": "f4e39e4f-715f-4e75-a247-c1f9fd0ee98c",
      "status": "Failed",
      "startTime": "2026-09-23T20:49:22.499172304Z",
      "endTime": "2026-09-23T21:18:28.082375104Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m40s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-hv5jpf95qtjp/cluster-candidate-4-21-xnt7bq should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f4a320>: 
    failed to create HCP cluster cluster-candidate-4-21-xnt7bq: failed waiting for cluster="cluster-candidate-4-21-xnt7bq" in resourcegroup="rg-candidate-4-21-hv5jpf95qtjp" to finish creating: GET https://rp.aro-hcp-dev-shard1-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4e39e4f-715f-4e75-a247-c1f9fd0ee98c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/f4e39e4f-715f-4e75-a247-c1f9fd0ee98c",
      "name": "f4e39e4f-715f-4e75-a247-c1f9fd0ee98c",
      "status": "Failed",
      "startTime": "2026-09-23T20:49:22.499172304Z",
      "endTime": "2026-09-23T21:18:28.082375104Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m40s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 7 time(s) (1 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T21:01:37Z
  Ended:       2026-09-23T21:30:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j9340032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t0uqkogi5je8dvqpd30ug1p31mmvf4h-a9d6f3s9m6a1f1a", pod="router-85d7d5999-qx49z", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2t0uqkogi5je8dvqpd30ug1p31mmvf4h-a9d6f3s9m6a1f1a/router-85d7d5999-qx49z has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T20:59:04Z
  Ended:       2026-09-23T21:31:03Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9340032-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9340032-svc).

alert fired 4 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T21:06:19Z
  Ended:       2026-09-23T21:21:20Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9340032-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 2:
  State:       Resolved
  Started:     2026-09-23T21:06:19Z
  Ended:       2026-09-23T21:36:17Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9340032-svc", component="backend", name="csstatedump", region="canadacentral", severity="warning"
  Description: Backend controller workqueue csstatedump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 3:
  State:       Resolved
  Started:     2026-09-23T21:06:19Z
  Ended:       2026-09-23T21:36:17Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9340032-svc", component="backend", name="datadump", region="canadacentral", severity="warning"
  Description: Backend controller workqueue datadump has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
Firing 4:
  State:       Resolved
  Started:     2026-09-23T21:06:19Z
  Ended:       2026-09-23T21:21:20Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j9340032-svc", component="backend", name="clusterclusterserviceupdatedispatch", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterclusterserviceupdatedispatch has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

time=2026-09-23T20:05:19.475Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 description="Step cluster-create\n  Kind: Shell\n  Command: ./aks-cluster-create\n"
time=2026-09-23T20:10:16.316Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=cluster-create stamp=2 err="stamp 2: error running Shell Step, failed to execute shell command: {\"time\":\"2026-09-23T20:05:19.482013132Z\",\"level\":\"INFO\",\"msg\":\"starting cluster reconciliation\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5982976-mgmt-2\",\"clusterName\":\"ci01-j5982976-mgmt-2\",\"region\":\"westus3\"}\n{\"time\":\"2026-09-23T20:05:21.259341341Z\",\"level\":\"INFO\",\"msg\":\"configured pools\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5982976-mgmt-2\",\"clusterName\":\"ci01-j5982976-mgmt-2\",\"region\":\"westus3\",\"system\":1,\"worker\":3,\"infra\":2}\n{\"time\":\"2026-09-23T20:05:21.679397764Z\",\"level\":\"INFO\",\"msg\":\"deploying configured resources\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5982976-mgmt-2\",\"clusterName\":\"ci01-j5982976-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"poolCount\":6,\"clusterFields\":null}\n{\"time\":\"2026-09-23T20:10:16.314281955Z\",\"level\":\"ERROR\",\"msg\":\"collecting deployment operations\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5982976-mgmt-2\",\"clusterName\":\"ci01-j5982976-mgmt-2\",\"region\":\"westus3\",\"deployment\":\"aks-cluster-create\",\"err\":\"Get \\\"https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5982976-mgmt-2/deployments/aks-cluster-create/operations?api-version=2025-04-01\\\": stream error: stream ID 41; CANCEL; received from peer\"}\n{\"time\":\"2026-09-23T20:10:16.314392598Z\",\"level\":\"ERROR\",\"msg\":\"run failed\",\"component\":\"aks-cluster-create\",\"subscriptionID\":\"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX\",\"resourceGroup\":\"hcp-underlay-ci01-j5982976-mgmt-2\",\"clusterName\":\"ci01-j5982976-mgmt-2\",\"region\":\"westus3\",\"err\":\"waiting for deployment aks-cluster-create: Get \\\"https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j5982976-mgmt-2/providers/Microsoft.Resources/deployments/aks-cluster-create/operationStatuses/08584114129632610784?api-version=2025-04-01&t=639257907264738192&c=MIIIJzCCBw-gAwIBAgIRAKCxVbScFDiCR8oT6qcGYJMwDQYJKoZIhvcNAQELBQAwNjE0MDIGA1UEAxMrQ0NNRSBHMSBUTFMgUlNBIDIwNDggU0hBMjU2IDIwNDkgV0NVUyBDQSAwMTAeFw0yNjA5MTkwMDU4MzJaFw0yNjEyMTQwNjU4MzJaMEAxPjA8BgNVBAMTNWFzeW5jb3BlcmF0aW9uc2lnbmluZ2NlcnRpZmljYXRlLm1hbmFnZW1lbnQuYXp1cmUuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAzFNlTnLezYuVV9JBgdhqA91jRJUtR4AjUkUKP5Nk7STlDt0_Hwx2OHYvMcuSX2B7klrS1nxF26zm4lnQKcCJTslJYfDrx756nLb5iGS6OS0koE0rze5-sUiu_IHHt28tpGv_9lvJvRWPDZc1uYCzrGyBpz6HvBzVgl3BeyWo9RTZYJhgBVNIATJuDRUzovVt8XczDob4euvsqOVr7rBx6wvKf79ASaeXB6PbT19HGywTrd9Xkd33sWfzQu5ZuHtSgX3i1sp_4BVE8VTuCkqIku5kzbulG3jvlSrsOmn680vsH3sS1APY1L7SymeuyNuVIv20HZmQukmjreOzZu-iGQIDAQABo4IFJDCCBSAwgZ0GA1UdIASBlTCBkjAMBgorBgEEAYI3ewEBMGYGCisGAQQBgjd7AgIwWDBWBggrBgEFBQcCAjBKHkgAMwAzAGUAMAAxADkAMgAxAC0ANABkADYANAAtADQAZgA4AGMALQBhADAANQA1AC0ANQBiAGQAYQBmAGYAZAA1AGUAMwAzAGQwDAYKKwYBBAGCN3sDAjAMBgorBgEEAYI3ewQCMAwGA1UdEwEB_wQCMAAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMA4GA1UdDwEB_wQEAwIFoDAdBgNVHQ4EFgQU7SRzDdTHXndOcavyYsWupdx4MecwHwYDVR0jBBgwFoAUFNI34PbWfX7djbq6ZasElCXglh0wggH7BgNVHR8EggHyMIIB7jB7oHmgd4Z1aHR0cDovL3ByaW1hcnktY2RuLnBraS5jb3JlLndpbmRvd3MubmV0L3dlc3RjZW50cmFsdXMvY3Jscy9jY21ld2VzdGNlbnRyYWx1c3BraS9jY21ld2VzdGNlbnRyYWx1c2ljYTAxLzQxL2N1cnJlbnQuY3JsMH2ge6B5hndodHRwOi8vc2Vjb25kYXJ5LWNkbi5wa2kuY29yZS53aW5kb3dzLm5ldC93ZXN0Y2VudHJhbHVzL2NybHMvY2NtZXdlc3RjZW50cmFsdXNwa2kvY2NtZXdlc3RjZW50cmFsdXNpY2EwMS80MS9jdXJyZW50LmNybDBsoGqgaIZmaHR0cDovL2NybC5taWNyb3NvZnQuY29tL3dlc3RjZW50cmFsdXMvY3Jscy9jY21ld2VzdGNlbnRyYWx1c3BraS9jY21ld2VzdGNlbnRyYWx1c2ljYTAxLzQxL2N1cnJlbnQuY3JsMIGBoH-gfYZ7aHR0cDovL2NjbWV3ZXN0Y2VudHJhbHVzcGtpLndlc3RjZW50cmFsdXMucGtpLmNvcmUud2luZG93cy5uZXQvY2VydGlmaWNhdGVBdXRob3JpdGllcy9jY21ld2VzdGNlbnRyYWx1c2ljYTAxLzQxL2N1cnJlbnQuY3JsMIICAAYIKwYBBQUHAQEEggHyMIIB7jB-BggrBgEFBQcwAoZyaHR0cDovL3ByaW1hcnktY2RuLnBraS5jb3JlLndpbmRvd3MubmV0L3dlc3RjZW50cmFsdXMvY2FjZXJ0cy9jY21ld2VzdGNlbnRyYWx1c3BraS9jY21ld2VzdGNlbnRyYWx1c2ljYTAxL2NlcnQuY2VyMIGABggrBgEFBQcwAoZ0aHR0cDovL3NlY29uZGFyeS1jZG4ucGtpLmNvcmUud2luZG93cy5uZXQvd2VzdGNlbnRyYWx1cy9jYWNlcnRzL2NjbWV3ZXN0Y2VudHJhbHVzcGtpL2NjbWV3ZXN0Y2VudHJhbHVzaWNhMDEvY2VydC5jZXIwbwYIKwYBBQUHMAKGY2h0dHA6Ly9jcmwubWljcm9zb2Z0LmNvbS93ZXN0Y2VudHJhbHVzL2NhY2VydHMvY2NtZXdlc3RjZW50cmFsdXNwa2kvY2NtZXdlc3RjZW50cmFsdXNpY2EwMS9jZXJ0LmNlcjB4BggrBgEFBQcwAoZsaHR0cDovL2NjbWV3ZXN0Y2VudHJhbHVzcGtpLndlc3RjZW50cmFsdXMucGtpLmNvcmUud2luZG93cy5uZXQvY2VydGlmaWNhdGVBdXRob3JpdGllcy9jY21ld2VzdGNlbnRyYWx1c2ljYTAxMA0GCSqGSIb3DQEBCwUAA4IBAQA5ody_Abz9ik5_3WVrjOb7e5cyPkeuDRHrpQ3VO0TeurG5LwTtQ-Yu-lE3ow2Qq4IbIIP6kpzWMQ_woGCp96pGdwvp7vLu4SYc-Rp-CKqLpJtXBRQ01RDbqBZWPwrrgrfMjTaopq1rEEHOBBqt2X4fPXlf91DULpbYJikwKVt4rSQPFgvzjtIpRSeHkIIw8oA95SBfgvs-CIisoc1h-1J1s6jmYw8Kqgqc-NoVLrvf4qRHrd8_0YK2nmtl5XnHD8a50gNVozHfx52l7NfwV_tYfwIO82lHrNxrpLkb7gSAAoAkma6s2Tjh23fpSj-E9OFOFMeuCwnaykbAggIZp9hc&s=xAFEUIMkN7HqEcaWnjPt98SPv2n8tDGNfsg9g3ETUj427eutsI2z-oSn-NYDEALQ9cMRKqky-VWvsQV0qp29lgMWk_hw47hsrPBP4zu666M_n4lLgU8QDrRKLDzrIYelxUd035HLMh9F5fWNHrGNulgbDCkeZeqOoc6EAne-535SWhie6nA952GikZdg_fD2pGBdL53D40Ycq2WqzPFAtYN4D-nKCFWKG8QoWd59PYeAW-C_ZWVMdATP9LmSEVVMHqMF13HK7-5WRYDFxs-LPKfY7okL22TjrA2RLKxuv1xWiSUxb0QLisSFa7Bqvbbl6syA1C3_0qKLeHs_XDklIA&h=dLBpH5niUgnkOleLXSR6xT4LrPMGJuQLdmK3gutCEoY\\\": stream error: stream ID 33; CANCEL; received from peer\",\"elapsed\":\"4m56.832387486s\"}\n exit status 1"

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T21:21:41Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j5527936-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-d7xzw", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-d7xzw (namespace arobit) on cluster ci01-j5527936-svc has averaged 125.9% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-ks79lqvdvdz4/cluster-candidate-4-21-tjs922 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001b96700>: 
    failed to create HCP cluster cluster-candidate-4-21-tjs922: failed waiting for cluster="cluster-candidate-4-21-tjs922" in resourcegroup="rg-candidate-4-21-ks79lqvdvdz4" to finish creating: GET https://rp.aro-hcp-dev-shard0-slot-02.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f164752-446b-48ce-a79b-2e5f890f02c6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f164752-446b-48ce-a79b-2e5f890f02c6",
      "name": "6f164752-446b-48ce-a79b-2e5f890f02c6",
      "status": "Failed",
      "startTime": "2026-09-23T20:08:29.367613866Z",
      "endTime": "2026-09-23T20:37:38.881566969Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, packageserver, olm-operator, ingress-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-ks79lqvdvdz4/cluster-candidate-4-21-tjs922 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001b96700>: 
    failed to create HCP cluster cluster-candidate-4-21-tjs922: failed waiting for cluster="cluster-candidate-4-21-tjs922" in resourcegroup="rg-candidate-4-21-ks79lqvdvdz4" to finish creating: GET https://rp.aro-hcp-dev-shard0-slot-02.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f164752-446b-48ce-a79b-2e5f890f02c6
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/6f164752-446b-48ce-a79b-2e5f890f02c6",
      "name": "6f164752-446b-48ce-a79b-2e5f890f02c6",
      "status": "Failed",
      "startTime": "2026-09-23T20:08:29.367613866Z",
      "endTime": "2026-09-23T20:37:38.881566969Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: ComponentsNotAvailable: Waiting for components to be available: cluster-network-operator, packageserver, olm-operator, ingress-operator, catalog-operator; hosted cluster degraded: UnavailableReplicas: [catalog-operator deployment has 1 unavailable replicas, cluster-network-operator deployment has 1 unavailable replicas, dns-operator deployment has 1 unavailable replicas, ingress-operator deployment has 1 unavailable replicas, olm-operator deployment has 1 unavailable replicas, packageserver deployment has 3 unavailable replicas]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T21:11:25Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j8964096-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-dhxkp", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-dhxkp (namespace arobit) on cluster ci01-j8964096-svc has averaged 124.8% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T20:25:47Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j4862464-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-6s587", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-6s587 (namespace arobit) on cluster ci01-j4862464-svc has averaged 125.2% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T19:49:29Z
  Ended:       2026-09-23T20:23:30Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j4862464-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j4862464-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000d7cd00>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-c9qhwvtrnfw9" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce602d9-c8f0-4f8b-9d73-4b32f73a4a7d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce602d9-c8f0-4f8b-9d73-4b32f73a4a7d",
      "name": "cce602d9-c8f0-4f8b-9d73-4b32f73a4a7d",
      "status": "Failed",
      "startTime": "2026-09-23T18:51:58.738258798Z",
      "endTime": "2026-09-23T19:01:09.161920644Z",
      "error": {
        "code": "InvalidResource",
        "message": "DataPlaneIdentitiesPermissionsValidation: MissingRequiredPermissions: Data plane operators missing required permissions: [{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-c9qhwvtrnfw9/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-20\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard2-03-14/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]},{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-c9qhwvtrnfw9/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-20/subnets/customer-vnet-subnet-4-20\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard2-03-14/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/read\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"},{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]}]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000d7cd00>: 
    failed waiting for cluster="cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-c9qhwvtrnfw9" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce602d9-c8f0-4f8b-9d73-4b32f73a4a7d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InvalidResource
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cce602d9-c8f0-4f8b-9d73-4b32f73a4a7d",
      "name": "cce602d9-c8f0-4f8b-9d73-4b32f73a4a7d",
      "status": "Failed",
      "startTime": "2026-09-23T18:51:58.738258798Z",
      "endTime": "2026-09-23T19:01:09.161920644Z",
      "error": {
        "code": "InvalidResource",
        "message": "DataPlaneIdentitiesPermissionsValidation: MissingRequiredPermissions: Data plane operators missing required permissions: [{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-c9qhwvtrnfw9/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-20\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard2-03-14/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]},{\"resource\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-c9qhwvtrnfw9/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-20/subnets/customer-vnet-subnet-4-20\",\"identity\":\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-dev-shard2-03-14/providers/Microsoft.ManagedIdentity/userAssignedIdentities/dp-image-registry\",\"decisions\":[{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/read\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"},{\"actionId\":\"Microsoft.Network/virtualNetworks/subnets/join/action\",\"isDataAction\":false,\"accessDecision\":\"NotAllowed\"}]}]"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc0006be420>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-4wt68pfrx42w" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/adb5be9c-18bf-41aa-925f-6eb547095403
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/adb5be9c-18bf-41aa-925f-6eb547095403",
      "name": "adb5be9c-18bf-41aa-925f-6eb547095403",
      "status": "Failed",
      "startTime": "2026-09-23T18:46:32.650200261Z",
      "endTime": "2026-09-23T19:05:38.929512243Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:177]: failed to create node pool np-1 for pull secret cluster
Unexpected error:
    <*fmt.wrapError | 0xc0006be420>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-4wt68pfrx42w" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/adb5be9c-18bf-41aa-925f-6eb547095403
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/adb5be9c-18bf-41aa-925f-6eb547095403",
      "name": "adb5be9c-18bf-41aa-925f-6eb547095403",
      "status": "Failed",
      "startTime": "2026-09-23T18:46:32.650200261Z",
      "endTime": "2026-09-23T19:05:38.929512243Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:357]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc00121bf40>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:37971->172.30.0.10:53: read: connection refused
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:357]: Timed out after 300.001s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc00121bf40>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.129.0.10:37971->172.30.0.10:53: read: connection refused
    ...

time=2026-09-23T18:34:52.490Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=svc-stamped step=deploy stamp=1 description="Step deploy\n Kind: Helm\n"
time=2026-09-23T18:34:52.977Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=svc-stamped step=deploy stamp=1 values="map[cloudEnvironmentName:AzurePublicCloud cosmos:map[name:arohcpci01-rp-j6376960 url:https://arohcpci01-rp-j6376960.documents.azure.com:443/] image:map[digest:sha256:6bc5ed31e42afccc2abd6b21b817cc402be829c20f8cc00e71fde01c9be1523a registry:arohcpsvcdev.azurecr.io repository:ci-op-6hlkxni0/pipeline] registration:map[aksResourceID:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6376960-mgmt-1/providers/Microsoft.ContainerService/managedClusters/ci01-j6376960-mgmt-1 autoApprove:true hostedClustersManagedIdentitiesKeyVaultURL:https://ah-ci01-mi-j6376960-1.vault.azure.net/ hostedClustersSecretsKeyVaultManagedIdentityClientID:b2cc94cd-3e8b-42a9-bf2a-e025837a95cd hostedClustersSecretsKeyVaultURL:https://ah-ci01-cx-j6376960-1.vault.azure.net/ kubeApplierCosmosContainerName:Manifests-MC-1 maestroConsumerName:hcp-underlay-aro-hcp-dev-shard2-slot-01-mgmt-1 maestroGRPCTarget:maestro-grpc.maestro.svc.cluster.local:8090 maestroRESTAPIURL:http://maestro.maestro.svc.cluster.local:8000 provisionShardStatus:active publicDNSZoneResourceID:/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j6376960/providers/Microsoft.Network/dnsZones/aro-hcp-dev-shard2-slot-01.hcp.osadev.cloud stampIdentifier:1] serviceAccountName:fleet]"
time=2026-09-23T18:34:52.977Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=svc-stamped step=deploy stamp=1
time=2026-09-23T18:34:54.319Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=svc-stamped step=deploy stamp=1 namespace=fleet
time=2026-09-23T18:34:54.894Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=svc-stamped step=deploy stamp=1 dryRun=false
time=2026-09-23T18:34:54.894Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=svc-stamped step=deploy stamp=1
time=2026-09-23T18:35:05.401Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=svc-stamped step=deploy stamp=1 numVersions=0
time=2026-09-23T18:35:05.530Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=svc-stamped step=deploy stamp=1 err="\"fleet-reg-ci01-j6376960-mgmt-1\" has no deployed releases"
time=2026-09-23T18:35:05.530Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Fleet.Registration resourceGroup=svc-stamped step=deploy stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: \"fleet-reg-ci01-j6376960-mgmt-1\" has no deployed releases"

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T18:53:46Z
  Ended:       2026-09-23T19:08:44Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j5312512-svc", component="frontend", method="delete", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: More than 5% of frontend requests for delete /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j5312512-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:346]: failed to uncordon node t9c6r3e7k2o1w9o-cordon-np-st9vc-2hvv8
Unexpected error:
    <*errors.StatusError | 0xc000ca99a0>: 
    Operation cannot be fulfilled on nodes "t9c6r3e7k2o1w9o-cordon-np-st9vc-2hvv8": the object has been modified; please apply your changes to the latest version and try again
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:346]: failed to uncordon node t9c6r3e7k2o1w9o-cordon-np-st9vc-2hvv8
Unexpected error:
    <*errors.StatusError | 0xc000ca99a0>: 
    Operation cannot be fulfilled on nodes "t9c6r3e7k2o1w9o-cordon-np-st9vc-2hvv8": the object has been modified; please apply your changes to the latest version and try again
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc000a21860>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-v9rkf4k56z2c" to finish creating: GET https://rp.aro-hcp-dev-shard1-slot-02.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/52906a0e-e465-4c26-861d-ab3698d37a5c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/52906a0e-e465-4c26-861d-ab3698d37a5c",
      "name": "52906a0e-e465-4c26-861d-ab3698d37a5c",
      "status": "Failed",
      "startTime": "2026-09-23T18:28:44.991686145Z",
      "endTime": "2026-09-23T18:47:50.145290668Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:138]: failed to create node pool
Expected success, but got an error:
    <*fmt.wrapError | 0xc000a21860>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "ingress-cert-le" in resourcegroup="ingress-cert-le-v9rkf4k56z2c" to finish creating: GET https://rp.aro-hcp-dev-shard1-slot-02.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/52906a0e-e465-4c26-861d-ab3698d37a5c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/52906a0e-e465-4c26-861d-ab3698d37a5c",
      "name": "52906a0e-e465-4c26-861d-ab3698d37a5c",
      "status": "Failed",
      "startTime": "2026-09-23T18:28:44.991686145Z",
      "endTime": "2026-09-23T18:47:50.145290668Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T19:18:05Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j4456960-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-ltf9x", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-ltf9x (namespace arobit) on cluster ci01-j4456960-svc has averaged 125% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T18:28:51Z
  Ended:       2026-09-23T19:02:51Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j4456960-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j4456960-svc).

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T18:30:41Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j9325440-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-nc4vq", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-nc4vq (namespace arobit) on cluster ci01-j9325440-svc has averaged 124% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000b97da0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-8g5rnldzw776" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/607fa875-dbd2-4396-9201-8fdd255f0980
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/607fa875-dbd2-4396-9201-8fdd255f0980",
      "name": "607fa875-dbd2-4396-9201-8fdd255f0980",
      "status": "Failed",
      "startTime": "2026-09-23T17:42:05.950445215Z",
      "endTime": "2026-09-23T18:01:06.854588111Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000b97da0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-8g5rnldzw776" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/607fa875-dbd2-4396-9201-8fdd255f0980
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/607fa875-dbd2-4396-9201-8fdd255f0980",
      "name": "607fa875-dbd2-4396-9201-8fdd255f0980",
      "status": "Failed",
      "startTime": "2026-09-23T17:42:05.950445215Z",
      "endTime": "2026-09-23T18:01:06.854588111Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T17:51:27Z
  Ended:       2026-09-23T18:00:24Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1331712-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1331712-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc001018240>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-57j9pfs2vmmr" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-00.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d3cf93e-ed99-43b6-aff0-026b8023aa5b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d3cf93e-ed99-43b6-aff0-026b8023aa5b",
      "name": "1d3cf93e-ed99-43b6-aff0-026b8023aa5b",
      "status": "Failed",
      "startTime": "2026-09-23T17:11:52.309512854Z",
      "endTime": "2026-09-23T17:40:54.426586313Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m20s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc001018240>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-57j9pfs2vmmr" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-00.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d3cf93e-ed99-43b6-aff0-026b8023aa5b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1d3cf93e-ed99-43b6-aff0-026b8023aa5b",
      "name": "1d3cf93e-ed99-43b6-aff0-026b8023aa5b",
      "status": "Failed",
      "startTime": "2026-09-23T17:11:52.309512854Z",
      "endTime": "2026-09-23T17:40:54.426586313Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 23m20s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T17:46:48Z
  Ended:       2026-09-23T17:57:49Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j9858944-mgmt-2", component="service-memory", container="kube-applier", namespace="kube-applier", pod="kube-applier-6f4c9cc744-8nwkb", severity="warning", team="hcp-sl"
  Description: Container kube-applier in pod kube-applier-6f4c9cc744-8nwkb (namespace kube-applier) on cluster ci01-j9858944-mgmt-2 has averaged 126.1% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:339]: failed to retrieve serial console logs for VM "s0t7n3b8m8p4t6a-worker-pfp6w-rth7j"
Unexpected error:
    <*errors.errorString | 0xc000a9b5e0>: 
    expected status 200 OK, got 500: {
        "error": {
            "code": "InternalServerError",
            "message": "Internal server error."
        }
    }
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:339]: failed to retrieve serial console logs for VM "s0t7n3b8m8p4t6a-worker-pfp6w-rth7j"
Unexpected error:
    <*errors.errorString | 0xc000a9b5e0>: 
    expected status 200 OK, got 500: {
        "error": {
            "code": "InternalServerError",
            "message": "Internal server error."
        }
    }
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc000f1c4c0>: 
    DNS for route host agnhost-e2e-sample-app-7fjnt.apps.aro.v20260901.jogf.aro-hcp-dev-shard1-slot-04.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7fjnt.apps.aro.v20260901.jogf.aro-hcp-dev-shard1-slot-04.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7fjnt.apps.aro.v20260901.jogf.aro-hcp-dev-shard1-slot-04.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:121]: failed to verify simple web app runs on v20260901preview cluster "v20260901"
Unexpected error:
    <*fmt.wrapError | 0xc000f1c4c0>: 
    DNS for route host agnhost-e2e-sample-app-7fjnt.apps.aro.v20260901.jogf.aro-hcp-dev-shard1-slot-04.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-7fjnt.apps.aro.v20260901.jogf.aro-hcp-dev-shard1-slot-04.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-7fjnt.apps.aro.v20260901.jogf.aro-hcp-dev-shard1-slot-04.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T16:33:04Z
  Ended:       2026-09-23T16:39:07Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4918656-svc", component="backend", name="clustervalidationcontrolplaneidentitiespermissionsclustervalidation", region="westus3", severity="warning"
  Description: Backend controller workqueue clustervalidationcontrolplaneidentitiespermissionsclustervalidation has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:147]: Timed out after 900.006s.
not all KSM metrics appeared in Azure Monitor
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:143 with:
expected ingresscontroller_info metrics from kube-state-metrics but got no results
Expected
    <[]prometheus.Result | len:0, cap:0>: ...
not to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:147]: Timed out after 900.006s.
not all KSM metrics appeared in Azure Monitor
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/ksm_hcp_metrics.go:143 with:
expected ingresscontroller_info metrics from kube-state-metrics but got no results
Expected
    <[]prometheus.Result | len:0, cap:0>: ...
not to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc0018bec90>: 
    GET https://rp.aro-hcp-dev-shard3-slot-02.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/54444a02-0c9c-4321-935a-b1997c03724e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/54444a02-0c9c-4321-935a-b1997c03724e",
      "name": "54444a02-0c9c-4321-935a-b1997c03724e",
      "status": "Failed",
      "startTime": "2026-09-23T15:55:02.945988001Z",
      "endTime": "2026-09-23T16:24:05.016015435Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc0018bec90>: 
    GET https://rp.aro-hcp-dev-shard3-slot-02.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/54444a02-0c9c-4321-935a-b1997c03724e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/54444a02-0c9c-4321-935a-b1997c03724e",
      "name": "54444a02-0c9c-4321-935a-b1997c03724e",
      "status": "Failed",
      "startTime": "2026-09-23T15:55:02.945988001Z",
      "endTime": "2026-09-23T16:24:05.016015435Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:155]: failed to scale up node pool np-update-NODES from 2 to 3 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc0013ee480>: 
    failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-bq2n97vrj5tz" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-NODES in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-bq2n97vrj5tz, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:155]: failed to scale up node pool np-update-NODES from 2 to 3 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc0013ee480>: 
    failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-bq2n97vrj5tz" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-NODES in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-bq2n97vrj5tz, error: context deadline exceeded
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T16:15:00Z
  Ended:       2026-09-23T16:46:01Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0621440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0621440/providers/microsoft.monitor/accounts/hcps-j0621440", namespace="ocm-arohcpci01-2t0qglpi4c0a10c27inccv2cpdf94jj4-e6q8x7z0j2j9n3s", pod="arohcp-monitor-kube-state-metrics-7955558448-jcrjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t0qglpi4c0a10c27inccv2cpdf94jj4-e6q8x7z0j2j9n3s/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-23T16:15:00Z
  Ended:       2026-09-23T16:45:01Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j0621440-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0621440/providers/microsoft.monitor/accounts/hcps-j0621440", namespace="ocm-arohcpci01-2t0qglpi4c0a10c27inccv2cpdf94jj4-e6q8x7z0j2j9n3s", pod="arohcp-monitor-kube-state-metrics-7955558448-jcrjw", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t0qglpi4c0a10c27inccv2cpdf94jj4-e6q8x7z0j2j9n3s/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T16:17:56Z
  Ended:       2026-09-23T16:44:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0621440-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0621440/providers/microsoft.monitor/accounts/hcps-j0621440", namespace="ocm-arohcpci01-2t0qglpi4c0a10c27inccv2cpdf94jj4-e6q8x7z0j2j9n3s", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="b0ac0532-43d4-43f8-be90-7de951bc2ef5"
  Description: Pod ocm-arohcpci01-2t0qglpi4c0a10c27inccv2cpdf94jj4-e6q8x7z0j2j9n3s/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").
Firing 2:
  State:       Resolved
  Started:     2026-09-23T16:17:56Z
  Ended:       2026-09-23T16:45:56Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j0621440-mgmt-2", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.64.95:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0621440/providers/microsoft.monitor/accounts/hcps-j0621440", namespace="ocm-arohcpci01-2t0qglpi4c0a10c27inccv2cpdf94jj4-e6q8x7z0j2j9n3s", pod="etcd-2", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="CrashLoopBackOff", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="b0ac0532-43d4-43f8-be90-7de951bc2ef5"
  Description: Pod ocm-arohcpci01-2t0qglpi4c0a10c27inccv2cpdf94jj4-e6q8x7z0j2j9n3s/etcd-2 (etcd) is in waiting state (reason: "CrashLoopBackOff").

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--q72fthqnzbpj/cluster-hs-pre-hbzxm7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001b10260>: 
    failed to create HCP cluster cluster-hs-pre-hbzxm7: failed waiting for cluster="cluster-hs-pre-hbzxm7" in resourcegroup="rg-hs-pre--q72fthqnzbpj" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e32709e9-bd3d-4b8f-99d7-ef1867b3765b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e32709e9-bd3d-4b8f-99d7-ef1867b3765b",
      "name": "e32709e9-bd3d-4b8f-99d7-ef1867b3765b",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:45.68921584Z",
      "endTime": "2026-09-23T16:22:46.778697738Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_hypershift_presubmit.go:83]: HCP cluster rg-hs-pre--q72fthqnzbpj/cluster-hs-pre-hbzxm7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001b10260>: 
    failed to create HCP cluster cluster-hs-pre-hbzxm7: failed waiting for cluster="cluster-hs-pre-hbzxm7" in resourcegroup="rg-hs-pre--q72fthqnzbpj" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e32709e9-bd3d-4b8f-99d7-ef1867b3765b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/e32709e9-bd3d-4b8f-99d7-ef1867b3765b",
      "name": "e32709e9-bd3d-4b8f-99d7-ef1867b3765b",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:45.68921584Z",
      "endTime": "2026-09-23T16:22:46.778697738Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-jmdbzj" with version 4.21.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0007a0980>: 
    failed to create HCP cluster cluster-zstream-4-21-jmdbzj: failed waiting for cluster="cluster-zstream-4-21-jmdbzj" in resourcegroup="rg-zstream-upgrade-4-21-4gvzx8nqmkwd" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fcddd5b-c1a1-4e97-9499-1f7660ea745f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fcddd5b-c1a1-4e97-9499-1f7660ea745f",
      "name": "6fcddd5b-c1a1-4e97-9499-1f7660ea745f",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:47.347085548Z",
      "endTime": "2026-09-23T16:22:56.388404215Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-jmdbzj" with version 4.21.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0007a0980>: 
    failed to create HCP cluster cluster-zstream-4-21-jmdbzj: failed waiting for cluster="cluster-zstream-4-21-jmdbzj" in resourcegroup="rg-zstream-upgrade-4-21-4gvzx8nqmkwd" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fcddd5b-c1a1-4e97-9499-1f7660ea745f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/6fcddd5b-c1a1-4e97-9499-1f7660ea745f",
      "name": "6fcddd5b-c1a1-4e97-9499-1f7660ea745f",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:47.347085548Z",
      "endTime": "2026-09-23T16:22:56.388404215Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc00055a3c0>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-z85zj8vh8z7t" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fe9c27f3-12ab-47fb-ab19-f97558746925
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fe9c27f3-12ab-47fb-ab19-f97558746925",
      "name": "fe9c27f3-12ab-47fb-ab19-f97558746925",
      "status": "Failed",
      "startTime": "2026-09-23T15:54:38.637505727Z",
      "endTime": "2026-09-23T16:23:46.403891132Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc00055a3c0>: 
    failed to create HCP cluster cidr-connectivity-test: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-z85zj8vh8z7t" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fe9c27f3-12ab-47fb-ab19-f97558746925
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/fe9c27f3-12ab-47fb-ab19-f97558746925",
      "name": "fe9c27f3-12ab-47fb-ab19-f97558746925",
      "status": "Failed",
      "startTime": "2026-09-23T15:54:38.637505727Z",
      "endTime": "2026-09-23T16:23:46.403891132Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-t2gnbd" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000bfe480>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-t2gnbd: failed waiting for cluster="cp-ystream-upgrade-4-21-t2gnbd" in resourcegroup="rg-cp-ystream-upgrade-4-21-cbnzwcvqnmsl" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5e27dfe4-3862-4b07-8ae4-c4377dbf948f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5e27dfe4-3862-4b07-8ae4-c4377dbf948f",
      "name": "5e27dfe4-3862-4b07-8ae4-c4377dbf948f",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:52.790630075Z",
      "endTime": "2026-09-23T16:22:56.599612998Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-t2gnbd" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000bfe480>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-t2gnbd: failed waiting for cluster="cp-ystream-upgrade-4-21-t2gnbd" in resourcegroup="rg-cp-ystream-upgrade-4-21-cbnzwcvqnmsl" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5e27dfe4-3862-4b07-8ae4-c4377dbf948f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5e27dfe4-3862-4b07-8ae4-c4377dbf948f",
      "name": "5e27dfe4-3862-4b07-8ae4-c4377dbf948f",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:52.790630075Z",
      "endTime": "2026-09-23T16:22:56.599612998Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0005383e0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-lmfj4bczvz8s" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b9d58f-8270-477f-aa5e-50f2c6683b01
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b9d58f-8270-477f-aa5e-50f2c6683b01",
      "name": "13b9d58f-8270-477f-aa5e-50f2c6683b01",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:45.078676518Z",
      "endTime": "2026-09-23T16:22:46.814369908Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0005383e0>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-lmfj4bczvz8s" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b9d58f-8270-477f-aa5e-50f2c6683b01
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/13b9d58f-8270-477f-aa5e-50f2c6683b01",
      "name": "13b9d58f-8270-477f-aa5e-50f2c6683b01",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:45.078676518Z",
      "endTime": "2026-09-23T16:22:46.814369908Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-hj788x" with version 4.22.14 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000f801c0>: 
    failed to create HCP cluster cluster-zstream-4-22-hj788x: failed waiting for cluster="cluster-zstream-4-22-hj788x" in resourcegroup="rg-zstream-upgrade-4-22-jdz9qkrk7g87" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6ee9cf-b4cb-447c-b859-9eb837187023
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6ee9cf-b4cb-447c-b859-9eb837187023",
      "name": "cb6ee9cf-b4cb-447c-b859-9eb837187023",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:45.267758855Z",
      "endTime": "2026-09-23T16:22:46.837301028Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-hj788x" with version 4.22.14 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000f801c0>: 
    failed to create HCP cluster cluster-zstream-4-22-hj788x: failed waiting for cluster="cluster-zstream-4-22-hj788x" in resourcegroup="rg-zstream-upgrade-4-22-jdz9qkrk7g87" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6ee9cf-b4cb-447c-b859-9eb837187023
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/cb6ee9cf-b4cb-447c-b859-9eb837187023",
      "name": "cb6ee9cf-b4cb-447c-b859-9eb837187023",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:45.267758855Z",
      "endTime": "2026-09-23T16:22:46.837301028Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000cbc500>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kb9nfmvpzmj4" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f138583-3da6-4491-9246-0255d290efc1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f138583-3da6-4491-9246-0255d290efc1",
      "name": "0f138583-3da6-4491-9246-0255d290efc1",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:52.301957583Z",
      "endTime": "2026-09-23T16:22:56.376646542Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000cbc500>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-kb9nfmvpzmj4" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f138583-3da6-4491-9246-0255d290efc1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/0f138583-3da6-4491-9246-0255d290efc1",
      "name": "0f138583-3da6-4491-9246-0255d290efc1",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:52.301957583Z",
      "endTime": "2026-09-23T16:22:56.376646542Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-8w2jrzrjz9pw/cluster-candidate-4-20-wj5jwq should provision
Unexpected error:
    <*fmt.wrapError | 0xc001648460>: 
    failed to create HCP cluster cluster-candidate-4-20-wj5jwq: failed waiting for cluster="cluster-candidate-4-20-wj5jwq" in resourcegroup="rg-candidate-4-20-8w2jrzrjz9pw" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/52fba8b4-23a6-4130-89f0-5fc7e90a72f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/52fba8b4-23a6-4130-89f0-5fc7e90a72f0",
      "name": "52fba8b4-23a6-4130-89f0-5fc7e90a72f0",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:34.986564037Z",
      "endTime": "2026-09-23T16:22:36.376773532Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-8w2jrzrjz9pw/cluster-candidate-4-20-wj5jwq should provision
Unexpected error:
    <*fmt.wrapError | 0xc001648460>: 
    failed to create HCP cluster cluster-candidate-4-20-wj5jwq: failed waiting for cluster="cluster-candidate-4-20-wj5jwq" in resourcegroup="rg-candidate-4-20-8w2jrzrjz9pw" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/52fba8b4-23a6-4130-89f0-5fc7e90a72f0
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/52fba8b4-23a6-4130-89f0-5fc7e90a72f0",
      "name": "52fba8b4-23a6-4130-89f0-5fc7e90a72f0",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:34.986564037Z",
      "endTime": "2026-09-23T16:22:36.376773532Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000f646a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-cpzrksbzwbnx" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3e9be050-bf9c-42a3-967b-1db23fa7c39a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3e9be050-bf9c-42a3-967b-1db23fa7c39a",
      "name": "3e9be050-bf9c-42a3-967b-1db23fa7c39a",
      "status": "Failed",
      "startTime": "2026-09-23T15:55:20.371775923Z",
      "endTime": "2026-09-23T16:24:26.21161142Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000f646a0>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-cpzrksbzwbnx" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3e9be050-bf9c-42a3-967b-1db23fa7c39a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3e9be050-bf9c-42a3-967b-1db23fa7c39a",
      "name": "3e9be050-bf9c-42a3-967b-1db23fa7c39a",
      "status": "Failed",
      "startTime": "2026-09-23T15:55:20.371775923Z",
      "endTime": "2026-09-23T16:24:26.21161142Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc0012448d0>: 
    GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c4f0afe4-8459-49e1-889d-631d8ffdb648
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c4f0afe4-8459-49e1-889d-631d8ffdb648",
      "name": "c4f0afe4-8459-49e1-889d-631d8ffdb648",
      "status": "Failed",
      "startTime": "2026-09-23T15:54:30.756948595Z",
      "endTime": "2026-09-23T16:23:36.211198776Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:131]: failed to wait for first cluster "basic-hcp-cluster" to complete creation (timeout '30.000000' minutes)
Unexpected error:
    <*exported.ResponseError | 0xc0012448d0>: 
    GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c4f0afe4-8459-49e1-889d-631d8ffdb648
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c4f0afe4-8459-49e1-889d-631d8ffdb648",
      "name": "c4f0afe4-8459-49e1-889d-631d8ffdb648",
      "status": "Failed",
      "startTime": "2026-09-23T15:54:30.756948595Z",
      "endTime": "2026-09-23T16:23:36.211198776Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b24340>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-pp5dtlw88rfs" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bf447353-9164-44e1-b7bb-b41146af7064
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bf447353-9164-44e1-b7bb-b41146af7064",
      "name": "bf447353-9164-44e1-b7bb-b41146af7064",
      "status": "Failed",
      "startTime": "2026-09-23T16:17:57.447240329Z",
      "endTime": "2026-09-23T16:47:00.702969424Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b24340>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-pp5dtlw88rfs" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bf447353-9164-44e1-b7bb-b41146af7064
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/bf447353-9164-44e1-b7bb-b41146af7064",
      "name": "bf447353-9164-44e1-b7bb-b41146af7064",
      "status": "Failed",
      "startTime": "2026-09-23T16:17:57.447240329Z",
      "endTime": "2026-09-23T16:47:00.702969424Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00107a1e0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-z5kd6wr2c86f" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5506d31e-96cb-4248-a2bc-e5375063105d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5506d31e-96cb-4248-a2bc-e5375063105d",
      "name": "5506d31e-96cb-4248-a2bc-e5375063105d",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:52.576887124Z",
      "endTime": "2026-09-23T16:22:56.469457208Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00107a1e0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed waiting for cluster="cordon-hcp-cluster" in resourcegroup="rg-cordon-test-z5kd6wr2c86f" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5506d31e-96cb-4248-a2bc-e5375063105d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/5506d31e-96cb-4248-a2bc-e5375063105d",
      "name": "5506d31e-96cb-4248-a2bc-e5375063105d",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:52.576887124Z",
      "endTime": "2026-09-23T16:22:56.469457208Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000d5cc20>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-8j8ndzvzp2dj" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a2be5479-1530-4944-82f9-a854036654b2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a2be5479-1530-4944-82f9-a854036654b2",
      "name": "a2be5479-1530-4944-82f9-a854036654b2",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:53.658130554Z",
      "endTime": "2026-09-23T16:22:56.230648382Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000d5cc20>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-8j8ndzvzp2dj" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a2be5479-1530-4944-82f9-a854036654b2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/a2be5479-1530-4944-82f9-a854036654b2",
      "name": "a2be5479-1530-4944-82f9-a854036654b2",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:53.658130554Z",
      "endTime": "2026-09-23T16:22:56.230648382Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-skmlmr with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc00166a640>: 
    failed to create HCP cluster np-version-upgrade-cluster-skmlmr: failed waiting for cluster="np-version-upgrade-cluster-skmlmr" in resourcegroup="rg-np-version-upgrade-skmlmr-gth9wdbm2lgd" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/edca57d8-1d8f-46a8-8263-c347e8c7a645
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/edca57d8-1d8f-46a8-8263-c347e8c7a645",
      "name": "edca57d8-1d8f-46a8-8263-c347e8c7a645",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:52.126846914Z",
      "endTime": "2026-09-23T16:22:56.455577909Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-skmlmr with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc00166a640>: 
    failed to create HCP cluster np-version-upgrade-cluster-skmlmr: failed waiting for cluster="np-version-upgrade-cluster-skmlmr" in resourcegroup="rg-np-version-upgrade-skmlmr-gth9wdbm2lgd" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/edca57d8-1d8f-46a8-8263-c347e8c7a645
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/edca57d8-1d8f-46a8-8263-c347e8c7a645",
      "name": "edca57d8-1d8f-46a8-8263-c347e8c7a645",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:52.126846914Z",
      "endTime": "2026-09-23T16:22:56.455577909Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test
Unexpected error:
    <*fmt.wrapError | 0xc0012263c0>: 
    failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-9v628njd5r4h" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eab71cd7-f7c6-4e17-a391-0a416ce9e57c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eab71cd7-f7c6-4e17-a391-0a416ce9e57c",
      "name": "eab71cd7-f7c6-4e17-a391-0a416ce9e57c",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:45.077341251Z",
      "endTime": "2026-09-23T16:22:46.744916423Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:308]: failed to create HCP cluster "sre-hcp-cluster-sc" for serial console test
Unexpected error:
    <*fmt.wrapError | 0xc0012263c0>: 
    failed to create HCP cluster sre-hcp-cluster-sc: failed waiting for cluster="sre-hcp-cluster-sc" in resourcegroup="admin-api-serialconsole-9v628njd5r4h" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eab71cd7-f7c6-4e17-a391-0a416ce9e57c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eab71cd7-f7c6-4e17-a391-0a416ce9e57c",
      "name": "eab71cd7-f7c6-4e17-a391-0a416ce9e57c",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:45.077341251Z",
      "endTime": "2026-09-23T16:22:46.744916423Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000f18e20>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-j694t442mj2r" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f6b220ce-3096-4a36-8499-22cf011cf0a8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f6b220ce-3096-4a36-8499-22cf011cf0a8",
      "name": "f6b220ce-3096-4a36-8499-22cf011cf0a8",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:53.255245987Z",
      "endTime": "2026-09-23T16:22:56.363291103Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000f18e20>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-j694t442mj2r" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f6b220ce-3096-4a36-8499-22cf011cf0a8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/f6b220ce-3096-4a36-8499-22cf011cf0a8",
      "name": "f6b220ce-3096-4a36-8499-22cf011cf0a8",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:53.255245987Z",
      "endTime": "2026-09-23T16:22:56.363291103Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-gt4scq at version 4.22.15
Unexpected error:
    <*fmt.wrapError | 0xc0018268c0>: 
    failed to create HCP cluster np-skip-minor-gt4scq: failed waiting for cluster="np-skip-minor-gt4scq" in resourcegroup="rg-np-skip-minor-gt4scq-f9nsllw59lzt" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/949773e8-36e8-45db-a78e-a47066415d62
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/949773e8-36e8-45db-a78e-a47066415d62",
      "name": "949773e8-36e8-45db-a78e-a47066415d62",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:50.566834881Z",
      "endTime": "2026-09-23T16:22:56.625731333Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-gt4scq at version 4.22.15
Unexpected error:
    <*fmt.wrapError | 0xc0018268c0>: 
    failed to create HCP cluster np-skip-minor-gt4scq: failed waiting for cluster="np-skip-minor-gt4scq" in resourcegroup="rg-np-skip-minor-gt4scq-f9nsllw59lzt" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/949773e8-36e8-45db-a78e-a47066415d62
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/949773e8-36e8-45db-a78e-a47066415d62",
      "name": "949773e8-36e8-45db-a78e-a47066415d62",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:50.566834881Z",
      "endTime": "2026-09-23T16:22:56.625731333Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-7gd6nm" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc0014f4660>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-7gd6nm: failed waiting for cluster="cp-ystream-upgrade-5-0-7gd6nm" in resourcegroup="rg-cp-ystream-upgrade-5-0-rrnk84sgmlwn" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77c395bf-c417-4455-9807-799065c6b12d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77c395bf-c417-4455-9807-799065c6b12d",
      "name": "77c395bf-c417-4455-9807-799065c6b12d",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:52.926344123Z",
      "endTime": "2026-09-23T16:22:56.209934575Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-7gd6nm" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc0014f4660>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-7gd6nm: failed waiting for cluster="cp-ystream-upgrade-5-0-7gd6nm" in resourcegroup="rg-cp-ystream-upgrade-5-0-rrnk84sgmlwn" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77c395bf-c417-4455-9807-799065c6b12d
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/77c395bf-c417-4455-9807-799065c6b12d",
      "name": "77c395bf-c417-4455-9807-799065c6b12d",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:52.926344123Z",
      "endTime": "2026-09-23T16:22:56.209934575Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc001164480>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-sg68hlrsd9c4" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c1a43583-7e84-4614-815f-0f428db0df68
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c1a43583-7e84-4614-815f-0f428db0df68",
      "name": "c1a43583-7e84-4614-815f-0f428db0df68",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:46.818708285Z",
      "endTime": "2026-09-23T16:22:46.588447682Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc001164480>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-sg68hlrsd9c4" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c1a43583-7e84-4614-815f-0f428db0df68
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c1a43583-7e84-4614-815f-0f428db0df68",
      "name": "c1a43583-7e84-4614-815f-0f428db0df68",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:46.818708285Z",
      "endTime": "2026-09-23T16:22:46.588447682Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc0010d4240>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-tj58cr2mq6ls" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eb238e5a-3a60-4002-b053-e96c3bdeb2ac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eb238e5a-3a60-4002-b053-e96c3bdeb2ac",
      "name": "eb238e5a-3a60-4002-b053-e96c3bdeb2ac",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:46.408421024Z",
      "endTime": "2026-09-23T16:22:46.864374297Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc0010d4240>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-tj58cr2mq6ls" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eb238e5a-3a60-4002-b053-e96c3bdeb2ac
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/eb238e5a-3a60-4002-b053-e96c3bdeb2ac",
      "name": "eb238e5a-3a60-4002-b053-e96c3bdeb2ac",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:46.408421024Z",
      "endTime": "2026-09-23T16:22:46.864374297Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d141e0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-9lh7k5m447kj" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02a3231c-7cad-4feb-a266-f06b18888bfe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02a3231c-7cad-4feb-a266-f06b18888bfe",
      "name": "02a3231c-7cad-4feb-a266-f06b18888bfe",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:50.505483407Z",
      "endTime": "2026-09-23T16:22:56.544110064Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d141e0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed waiting for cluster="np-delete-hcp-cluster" in resourcegroup="nodepool-delete-9lh7k5m447kj" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02a3231c-7cad-4feb-a266-f06b18888bfe
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/02a3231c-7cad-4feb-a266-f06b18888bfe",
      "name": "02a3231c-7cad-4feb-a266-f06b18888bfe",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:50.505483407Z",
      "endTime": "2026-09-23T16:22:56.544110064Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-5vk9ph" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc0017a2760>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-5vk9ph: failed waiting for cluster="cp-ystream-upgrade-4-22-5vk9ph" in resourcegroup="rg-cp-ystream-upgrade-4-22-dw4krrwmbz6s" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c6e1dafd-03ab-46f2-99d1-424bbb7b0f7c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c6e1dafd-03ab-46f2-99d1-424bbb7b0f7c",
      "name": "c6e1dafd-03ab-46f2-99d1-424bbb7b0f7c",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:52.801712787Z",
      "endTime": "2026-09-23T16:22:56.470110002Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-5vk9ph" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc0017a2760>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-5vk9ph: failed waiting for cluster="cp-ystream-upgrade-4-22-5vk9ph" in resourcegroup="rg-cp-ystream-upgrade-4-22-dw4krrwmbz6s" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c6e1dafd-03ab-46f2-99d1-424bbb7b0f7c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/c6e1dafd-03ab-46f2-99d1-424bbb7b0f7c",
      "name": "c6e1dafd-03ab-46f2-99d1-424bbb7b0f7c",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:52.801712787Z",
      "endTime": "2026-09-23T16:22:56.470110002Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Cluster provisioning failed

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000f98140>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-l95n8mx296wc" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7e3202c3-297e-4668-8e78-60c3e561dd70
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7e3202c3-297e-4668-8e78-60c3e561dd70",
      "name": "7e3202c3-297e-4668-8e78-60c3e561dd70",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:47.243780388Z",
      "endTime": "2026-09-23T16:22:56.457631227Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000f98140>: 
    failed to create HCP cluster v20260901: failed waiting for cluster="v20260901" in resourcegroup="v20260901preview-l95n8mx296wc" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7e3202c3-297e-4668-8e78-60c3e561dd70
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/7e3202c3-297e-4668-8e78-60c3e561dd70",
      "name": "7e3202c3-297e-4668-8e78-60c3e561dd70",
      "status": "Failed",
      "startTime": "2026-09-23T15:53:47.243780388Z",
      "endTime": "2026-09-23T16:22:56.457631227Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 13 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T16:16:22Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3938304-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3938304/providers/microsoft.monitor/accounts/services-j3938304", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7955558448-k26xd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-23T16:16:22Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3938304-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3938304/providers/microsoft.monitor/accounts/services-j3938304", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7955558448-k26xd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-23T16:27:22Z
  Ended:       2026-09-23T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3938304-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3938304/providers/microsoft.monitor/accounts/services-j3938304", namespace="klusterlet-2t0qg2he3nq1jn6qmjkr3aaspbgstbat", pod="arohcp-monitor-kube-state-metrics-7955558448-k26xd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t0qg2he3nq1jn6qmjkr3aaspbgstbat/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-23T16:27:22Z
  Ended:       2026-09-23T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3938304-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3938304/providers/microsoft.monitor/accounts/services-j3938304", namespace="klusterlet-2t0qg2h9j7mhi4jsf9pvdiin21umprdi", pod="arohcp-monitor-kube-state-metrics-7955558448-k26xd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t0qg2h9j7mhi4jsf9pvdiin21umprdi/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-23T16:27:22Z
  Ended:       2026-09-23T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3938304-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3938304/providers/microsoft.monitor/accounts/services-j3938304", namespace="klusterlet-2t0qg3radjabcqnahr40ucc01lfmk5a8", pod="arohcp-monitor-kube-state-metrics-7955558448-k26xd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t0qg3radjabcqnahr40ucc01lfmk5a8/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-23T16:27:22Z
  Ended:       2026-09-23T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3938304-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3938304/providers/microsoft.monitor/accounts/services-j3938304", namespace="klusterlet-2t0qge4i1v30v9fissqi7v4cphc1oevm", pod="arohcp-monitor-kube-state-metrics-7955558448-k26xd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t0qge4i1v30v9fissqi7v4cphc1oevm/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-23T16:27:22Z
  Ended:       2026-09-23T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3938304-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3938304/providers/microsoft.monitor/accounts/services-j3938304", namespace="klusterlet-2t0qg2uevkmvfsffrb6l8hl332p8ardq", pod="arohcp-monitor-kube-state-metrics-7955558448-k26xd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t0qg2uevkmvfsffrb6l8hl332p8ardq/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-23T16:27:22Z
  Ended:       2026-09-23T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3938304-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3938304/providers/microsoft.monitor/accounts/services-j3938304", namespace="klusterlet-2t0qg504tgg42v8ako2cd4g5d2nos1ka", pod="arohcp-monitor-kube-state-metrics-7955558448-k26xd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t0qg504tgg42v8ako2cd4g5d2nos1ka/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-23T16:27:22Z
  Ended:       2026-09-23T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3938304-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3938304/providers/microsoft.monitor/accounts/services-j3938304", namespace="klusterlet-2t0qg55h7r4jqvn6q2ve0nvli480gqgj", pod="arohcp-monitor-kube-state-metrics-7955558448-k26xd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t0qg55h7r4jqvn6q2ve0nvli480gqgj/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-23T16:27:22Z
  Ended:       2026-09-23T16:33:34Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3938304-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3938304/providers/microsoft.monitor/accounts/services-j3938304", namespace="klusterlet-2t0qg4fmbf9m1ge8flue3mi2da0mjinv", pod="arohcp-monitor-kube-state-metrics-7955558448-k26xd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t0qg4fmbf9m1ge8flue3mi2da0mjinv/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-23T16:27:22Z
  Ended:       2026-09-23T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3938304-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3938304/providers/microsoft.monitor/accounts/services-j3938304", namespace="klusterlet-2t0qg4ehbc8a4kaevorrib9n0fhaggup", pod="arohcp-monitor-kube-state-metrics-7955558448-k26xd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t0qg4ehbc8a4kaevorrib9n0fhaggup/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-23T16:27:22Z
  Ended:       2026-09-23T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3938304-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3938304/providers/microsoft.monitor/accounts/services-j3938304", namespace="klusterlet-2t0qge4qjfei7usbdki96o5frqdag2t4", pod="arohcp-monitor-kube-state-metrics-7955558448-k26xd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t0qge4qjfei7usbdki96o5frqdag2t4/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-23T16:27:22Z
  Ended:       2026-09-23T16:33:33Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j3938304-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3938304/providers/microsoft.monitor/accounts/services-j3938304", namespace="klusterlet-2t0qg2pu5vkhroo89ii97sbdd2h2evli", pod="arohcp-monitor-kube-state-metrics-7955558448-k26xd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment klusterlet-2t0qg2pu5vkhroo89ii97sbdd2h2evli/klusterlet-addon-workmgr has not matched the expected number of replicas for longer than 30 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T16:26:24Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j3938304-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3938304/providers/microsoft.monitor/accounts/services-j3938304", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7955558448-k26xd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-23T16:27:22Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentRolloutStuck", cluster="ci01-j3938304-mgmt-2", component="kubernetes-infrastructure", condition="Progressing", container="kube-state-metrics", deployment="klusterlet-addon-workmgr", endpoint="http", environment="ci01", instance="10.128.64.85:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3938304/providers/microsoft.monitor/accounts/services-j3938304", namespace="open-cluster-management-agent-addon", pod="arohcp-monitor-kube-state-metrics-7955558448-k26xd", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", reason="progressdeadlineexceeded", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", status="false"
  Description: Rollout of deployment open-cluster-management-agent-addon/klusterlet-addon-workmgr is not progressing for longer than 30 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T15:53:20Z
  Ended:       2026-09-23T15:59:18Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0727168-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0727168-svc).

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T16:11:32Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j5116032-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-cmbd9", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-cmbd9 (namespace arobit) on cluster ci01-j5116032-svc has averaged 125.3% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-pt8mvm"
Unexpected error:
    <*fmt.wrapError | 0xc00118c460>: 
    DNS for route host agnhost-e2e-sample-app-sv7vn.apps.aro.b3o6y9u3m3w9a8r.7oej.aro-hcp-dev-shard0-slot-00.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-sv7vn.apps.aro.b3o6y9u3m3w9a8r.7oej.aro-hcp-dev-shard0-slot-00.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv7vn.apps.aro.b3o6y9u3m3w9a8r.7oej.aro-hcp-dev-shard0-slot-00.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:189]: failed to verify simple web app runs on cluster "cluster-candidate-4-22-pt8mvm"
Unexpected error:
    <*fmt.wrapError | 0xc00118c460>: 
    DNS for route host agnhost-e2e-sample-app-sv7vn.apps.aro.b3o6y9u3m3w9a8r.7oej.aro-hcp-dev-shard0-slot-00.hcp.osadev.cloud did not resolve: DNS for agnhost-e2e-sample-app-sv7vn.apps.aro.b3o6y9u3m3w9a8r.7oej.aro-hcp-dev-shard0-slot-00.hcp.osadev.cloud did not resolve within 10m0s (last error: lookup agnhost-e2e-sample-app-sv7vn.apps.aro.b3o6y9u3m3w9a8r.7oej.aro-hcp-dev-shard0-slot-00.hcp.osadev.cloud on 172.30.0.10:53: no such host): context deadline exceeded
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T14:55:12Z
  Ended:       2026-09-23T15:28:14Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3326976-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3326976-svc).

time=2026-09-23T14:16:23.495Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=global step=mirror-image description="Step mirror-image\n  Kind: ImageMirror\n  From {<nil> clustersService.image.registry <nil>}:{<nil> clustersService.image.repository <nil>}@{<nil> clustersService.image.digest <nil>} to {<nil> acr.svc.name <nil>}\n"
time=2026-09-23T14:17:09.991Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.ClusterService resourceGroup=global step=mirror-image err="error running Image Mirror Step, failed to execute shell command: Checking USE_OC_LOGIN_REGISTRIES: registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org registry.build11.ci.openshift.org\nFetch pull secret for source registry quay.io from arohcpdev-global KV.\nLogging into target ACR arohcpsvcdev.\nLogin Succeeded\nMirroring image quay.io/app-sre/aro-hcp-clusters-service@sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6 to arohcpsvcdev.azurecr.io/app-sre/aro-hcp-clusters-service:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6.\nThe image will still be available under it's original digest sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6 in the target registry.\nError response from registry: failed to resolve sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: quay.io/app-sre/aro-hcp-clusters-service@sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: not found\nCommand failed (attempt 1/5). Retrying in 2s...\nError response from registry: failed to resolve sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: quay.io/app-sre/aro-hcp-clusters-service@sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: not found\nCommand failed (attempt 2/5). Retrying in 4s...\nError response from registry: failed to resolve sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: quay.io/app-sre/aro-hcp-clusters-service@sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: not found\nCommand failed (attempt 3/5). Retrying in 8s...\nError response from registry: failed to resolve sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: quay.io/app-sre/aro-hcp-clusters-service@sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: not found\nCommand failed (attempt 4/5). Retrying in 16s...\nError response from registry: failed to resolve sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: quay.io/app-sre/aro-hcp-clusters-service@sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6: not found\nCommand failed after 5 attempts: oras cp quay.io/app-sre/aro-hcp-clusters-service@sha256:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6 arohcpsvcdev.azurecr.io/app-sre/aro-hcp-clusters-service:299c08f05b5b37da0e9c8b727bd0f57e519da49d0bb6daf96b57d0744493e5a6 --from-registry-config /tmp/tmp.DLBEJUu7Ue/containers/auth.json --to-registry-config /tmp/tmp.DLBEJUu7Ue/containers/auth.json\n exit status 1"

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T15:27:01Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j1372160-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-kr2cs", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-kr2cs (namespace arobit) on cluster ci01-j1372160-svc has averaged 125.9% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000e2ede0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-2jl588bhfvmd" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8d975644-73c1-4964-98e7-311fe1b5f246
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8d975644-73c1-4964-98e7-311fe1b5f246",
      "name": "8d975644-73c1-4964-98e7-311fe1b5f246",
      "status": "Failed",
      "startTime": "2026-09-23T14:47:29.897134227Z",
      "endTime": "2026-09-23T15:06:35.791795552Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:220]: failed to create no-AZ nodepool autoscale-noaz with autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000e2ede0>: 
    failed to create NodePool autoscale-noaz: failed waiting for nodepool="autoscale-noaz" for cluster "np-autoscale-cluster" in resourcegroup="np-autoscaling-2jl588bhfvmd" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-01.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8d975644-73c1-4964-98e7-311fe1b5f246
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8d975644-73c1-4964-98e7-311fe1b5f246",
      "name": "8d975644-73c1-4964-98e7-311fe1b5f246",
      "status": "Failed",
      "startTime": "2026-09-23T14:47:29.897134227Z",
      "endTime": "2026-09-23T15:06:35.791795552Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc00094e860>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-nkx9tc" in resourcegroup="rg-np-skip-minor-nkx9tc-svtsnwjlpglh" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-00.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2b0016fd-fafe-4b72-8a66-7b759b311c62
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2b0016fd-fafe-4b72-8a66-7b759b311c62",
      "name": "2b0016fd-fafe-4b72-8a66-7b759b311c62",
      "status": "Failed",
      "startTime": "2026-09-23T14:30:28.639210211Z",
      "endTime": "2026-09-23T14:49:28.972470638Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc00094e860>: 
    failed to create NodePool nps-420-422: failed waiting for nodepool="nps-420-422" for cluster "np-skip-minor-nkx9tc" in resourcegroup="rg-np-skip-minor-nkx9tc-svtsnwjlpglh" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-00.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2b0016fd-fafe-4b72-8a66-7b759b311c62
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/2b0016fd-fafe-4b72-8a66-7b759b311c62",
      "name": "2b0016fd-fafe-4b72-8a66-7b759b311c62",
      "status": "Failed",
      "startTime": "2026-09-23T14:30:28.639210211Z",
      "endTime": "2026-09-23T14:49:28.972470638Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:433]: failed to create worker nodepool for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc0010240a0>: 
    failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-w2f9x55rndpg" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-00.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/269f9f7f-d982-42df-9b95-a4f0fcdc404f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/269f9f7f-d982-42df-9b95-a4f0fcdc404f",
      "name": "269f9f7f-d982-42df-9b95-a4f0fcdc404f",
      "status": "Failed",
      "startTime": "2026-09-23T14:21:22.155873772Z",
      "endTime": "2026-09-23T14:22:05.314078185Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_api.go:433]: failed to create worker nodepool for boot diagnostics test
Unexpected error:
    <*fmt.wrapError | 0xc0010240a0>: 
    failed to create NodePool worker: failed waiting for nodepool="worker" for cluster "sre-hcp-cluster-bootdiag" in resourcegroup="admin-api-serialconsole-bootdiag-w2f9x55rndpg" to finish creating: GET https://rp.aro-hcp-dev-shard3-slot-00.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/269f9f7f-d982-42df-9b95-a4f0fcdc404f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/269f9f7f-d982-42df-9b95-a4f0fcdc404f",
      "name": "269f9f7f-d982-42df-9b95-a4f0fcdc404f",
      "status": "Failed",
      "startTime": "2026-09-23T14:21:22.155873772Z",
      "endTime": "2026-09-23T14:22:05.314078185Z",
      "error": {
        "code": "InternalServerError",
        "message": "[clusterServiceNodePoolStatus] GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Compute/skus\n--------------------------------------------------------------------------------\nRESPONSE 429: 429 Too Many Requests\nERROR CODE: SubscriptionRequestsThrottled\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"SubscriptionRequestsThrottled\",\n    \"message\": \"Number of 'read' requests for subscription 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' actor '47f69502-0065-4d9a-b19b-d403e183d2f4' exceeded. Please try again after '10' seconds after additional tokens are available. Refer to https://aka.ms/arm-throttling for additional information.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T13:47:46Z
  Ended:       2026-09-23T14:19:44Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j7183104-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j7183104-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0010253a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-fpvkvnrfcfj2" to finish creating: GET https://rp.aro-hcp-dev-shard0-slot-04.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1688403e-4bb7-4250-94fc-34a807ea0027
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1688403e-4bb7-4250-94fc-34a807ea0027",
      "name": "1688403e-4bb7-4250-94fc-34a807ea0027",
      "status": "Failed",
      "startTime": "2026-09-23T13:47:50.262026457Z",
      "endTime": "2026-09-23T14:06:54.054886757Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:128]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc0010253a0>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-fpvkvnrfcfj2" to finish creating: GET https://rp.aro-hcp-dev-shard0-slot-04.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1688403e-4bb7-4250-94fc-34a807ea0027
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/1688403e-4bb7-4250-94fc-34a807ea0027",
      "name": "1688403e-4bb7-4250-94fc-34a807ea0027",
      "status": "Failed",
      "startTime": "2026-09-23T13:47:50.262026457Z",
      "endTime": "2026-09-23T14:06:54.054886757Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc0007ff9c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-dzgz54ntn7hc" to finish creating: GET https://rp.aro-hcp-dev-shard0-slot-04.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/184bf507-60e3-4d4c-91bb-42f27f255323
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/184bf507-60e3-4d4c-91bb-42f27f255323",
      "name": "184bf507-60e3-4d4c-91bb-42f27f255323",
      "status": "Failed",
      "startTime": "2026-09-23T13:43:16.227264536Z",
      "endTime": "2026-09-23T14:02:20.066830362Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:139]: failed to create nodepool "np-1" for autoscaling cluster
Unexpected error:
    <*fmt.wrapError | 0xc0007ff9c0>: 
    failed to create NodePool np-1: failed waiting for nodepool="np-1" for cluster "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-dzgz54ntn7hc" to finish creating: GET https://rp.aro-hcp-dev-shard0-slot-04.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/184bf507-60e3-4d4c-91bb-42f27f255323
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/184bf507-60e3-4d4c-91bb-42f27f255323",
      "name": "184bf507-60e3-4d4c-91bb-42f27f255323",
      "status": "Failed",
      "startTime": "2026-09-23T13:43:16.227264536Z",
      "endTime": "2026-09-23T14:02:20.066830362Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T14:09:57Z
  Ended:       2026-09-23T14:15:57Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6650496-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6650496-svc).

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T14:29:18Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j6650496-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-wqp86", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-wqp86 (namespace arobit) on cluster ci01-j6650496-svc has averaged 125% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:357]: Timed out after 300.003s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc001a366b0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.20:59854->172.30.0.10:53: read: connection refused
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:357]: Timed out after 300.003s.
ClusterIssuer should become Ready (ACME account registered)
Expected success, but got an error:
    <*errors.errorString | 0xc001a366b0>: 
    clusterIssuer not ready: reason=ErrRegisterACMEAccount message=Failed to register ACME account: Get "https://acme-staging-v02.api.letsencrypt.org/directory": dial tcp: lookup acme-staging-v02.api.letsencrypt.org on 172.30.0.10:53: read udp 10.128.0.20:59854->172.30.0.10:53: read: connection refused
    ...

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T13:20:33Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j6900864-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-st2mg", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-st2mg (namespace arobit) on cluster ci01-j6900864-svc has averaged 124.6% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T12:28:03Z
  Ended:       2026-09-23T13:01:03Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6900864-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6900864-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T12:40:58Z
  Ended:       2026-09-23T12:46:58Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1142016-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1142016-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T12:03:55Z
  Ended:       2026-09-23T12:37:55Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j2532480-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j2532480-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc00190deb0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:389]: Timed out after 600.002s.
Certificate should be issued by Let's Encrypt staging via ACME DNS-01
Expected success, but got an error:
    <*errors.errorString | 0xc00190deb0>: 
    certificate not ready: reason=DoesNotExist message=Issuing certificate as Secret does not exist
    ...

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T11:26:05Z
  Ended:       2026-09-23T11:37:13Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j7059968-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j7059968-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T10:53:08Z
  Ended:       2026-09-23T10:59:09Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j2996736-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j2996736-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T10:52:54Z
  Ended:       2026-09-23T10:58:56Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1762304-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1762304-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc001078700>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-tsmxlnchqjdn" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a99324f-ced3-4b58-b157-e5f91f1fdc43
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a99324f-ced3-4b58-b157-e5f91f1fdc43",
      "name": "5a99324f-ced3-4b58-b157-e5f91f1fdc43",
      "status": "Failed",
      "startTime": "2026-09-23T10:04:12.923013072Z",
      "endTime": "2026-09-23T10:33:15.61163171Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc001078700>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-tsmxlnchqjdn" to finish creating: GET https://rp.aro-hcp-dev-shard2-slot-03.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a99324f-ced3-4b58-b157-e5f91f1fdc43
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5a99324f-ced3-4b58-b157-e5f91f1fdc43",
      "name": "5a99324f-ced3-4b58-b157-e5f91f1fdc43",
      "status": "Failed",
      "startTime": "2026-09-23T10:04:12.923013072Z",
      "endTime": "2026-09-23T10:33:15.61163171Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T10:25:13Z
  Ended:       2026-09-23T10:52:09Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j7833216-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.79:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7833216/providers/microsoft.monitor/accounts/hcps-j7833216", namespace="ocm-arohcpci01-2t0lc7i6ae878vg6j5amjnqk5ucv64ji-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7955558448-bj599", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t0lc7i6ae878vg6j5amjnqk5ucv64ji-cilium-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-23T10:25:14Z
  Ended:       2026-09-23T10:54:08Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j7833216-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.64.79:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j7833216/providers/microsoft.monitor/accounts/hcps-j7833216", namespace="ocm-arohcpci01-2t0lc7i6ae878vg6j5amjnqk5ucv64ji-cilium-cluster", pod="arohcp-monitor-kube-state-metrics-7955558448-bj599", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2t0lc7i6ae878vg6j5amjnqk5ucv64ji-cilium-cluster/etcd has not matched the expected number of replicas for longer than 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T10:47:31Z
  Ended:       2026-09-23T10:53:30Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6677120-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6677120-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T09:00:54Z
  Ended:       2026-09-23T09:34:54Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j4854400-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j4854400-svc).

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T09:48:09Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j6558336-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-j66dp", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-j66dp (namespace arobit) on cluster ci01-j6558336-svc has averaged 125.4% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T08:33:51Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatency6h30m", cluster="ci01-j5147264-svc", component="slo", long_window="6h", region="westus3", severity="3", short_window="30m", slo="frontend-latency"
  Description: More than 6% of Frontend requests on cluster ci01-j5147264-svc exceeded 1s on both the 6h and 30m windows (6x burn of the 1% latency budget).

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_dataplane.go:304]: guest nodes were no longer unschedulable after upgrade on cluster "cp-ystream-dp-cordon-4-21-8ndsmf"
Expected success, but got an error:
    <*errors.errorString | 0xc0013005c0>: 
    expected all guest nodes to be unschedulable, schedulable=e6r1q1p6z7w9f4r-np-dataplane-wct59-fvjmv, e6r1q1p6z7w9f4r-np-dataplane-wct59-nqv5r; nodes: e6r1q1p6z7w9f4r-np-dataplane-fqlcw-4wrz9 ready=true unschedulable=true created=2026-09-23T07:52:57Z nodePool=e6r1q1p6z7w9f4r-np-dataplane; e6r1q1p6z7w9f4r-np-dataplane-wct59-fvjmv ready=true unschedulable=false created=2026-09-23T08:05:13Z nodePool=e6r1q1p6z7w9f4r-np-dataplane; e6r1q1p6z7w9f4r-np-dataplane-wct59-nqv5r ready=false unschedulable=false created=2026-09-23T08:11:10Z nodePool=e6r1q1p6z7w9f4r-np-dataplane
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade_dataplane.go:304]: guest nodes were no longer unschedulable after upgrade on cluster "cp-ystream-dp-cordon-4-21-8ndsmf"
Expected success, but got an error:
    <*errors.errorString | 0xc0013005c0>: 
    expected all guest nodes to be unschedulable, schedulable=e6r1q1p6z7w9f4r-np-dataplane-wct59-fvjmv, e6r1q1p6z7w9f4r-np-dataplane-wct59-nqv5r; nodes: e6r1q1p6z7w9f4r-np-dataplane-fqlcw-4wrz9 ready=true unschedulable=true created=2026-09-23T07:52:57Z nodePool=e6r1q1p6z7w9f4r-np-dataplane; e6r1q1p6z7w9f4r-np-dataplane-wct59-fvjmv ready=true unschedulable=false created=2026-09-23T08:05:13Z nodePool=e6r1q1p6z7w9f4r-np-dataplane; e6r1q1p6z7w9f4r-np-dataplane-wct59-nqv5r ready=false unschedulable=false created=2026-09-23T08:11:10Z nodePool=e6r1q1p6z7w9f4r-np-dataplane
    ...

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T08:47:31Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j3684224-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-z8dk5", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-z8dk5 (namespace arobit) on cluster ci01-j3684224-svc has averaged 125.5% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T08:11:35Z
  Ended:       2026-09-23T08:22:32Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3684224-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3684224-svc).

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T08:39:18Z
  Severity:    Sev3
  Labels:      alertname="userJourneyFrontendLatency6h30m", cluster="ci01-j3407488-svc", component="slo", long_window="6h", region="westus3", severity="3", short_window="30m", slo="frontend-latency"
  Description: More than 6% of Frontend requests on cluster ci01-j3407488-svc exceeded 1s on both the 6h and 30m windows (6x burn of the 1% latency budget).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T08:08:37Z
  Ended:       2026-09-23T08:22:37Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3407488-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: The median (p50) of frontend request latency for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} has exceeded 250ms over the past 30 minutes (cluster ci01-j3407488-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T08:11:20Z
  Ended:       2026-09-23T08:18:18Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j5172864-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j5172864-svc).

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T08:39:50Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j5172864-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-j4msz", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-j4msz (namespace arobit) on cluster ci01-j5172864-svc has averaged 124.7% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T06:11:33Z
  Ended:       2026-09-23T06:20:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j3290112-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2t0homk1jhbv7gpr72fmt0jahgksipbf-m8t7b7k5f6z0o9y", pod="community-operators-catalog-5f9d56fbbf-bvthw", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2t0homk1jhbv7gpr72fmt0jahgksipbf-m8t7b7k5f6z0o9y/community-operators-catalog-5f9d56fbbf-bvthw has been in a non-ready state for longer than 5 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T06:43:51Z
  Ended:       2026-09-23T06:49:51Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1547520-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1547520-svc).

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T03:16:05Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3482624-svc", component="frontend", method="get", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters", severity="warning"
  Description: The median (p50) of frontend request latency for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters has exceeded 250ms over the past 30 minutes (cluster ci01-j3482624-svc).
Firing 2:
  State:       Resolved
  Started:     2026-09-23T03:29:12Z
  Ended:       2026-09-23T03:35:13Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3482624-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3482624-svc).

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T03:10:35Z
  Ended:       2026-09-23T03:20:35Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j3482624-svc", component="backend", name="clusterresources", region="centralus", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.

alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-23T03:37:03Z
  Ended:       2026-09-23T03:47:04Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j3482624-svc", component="frontend", method="get", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters exceeded 1 second over the past 30 minutes (cluster ci01-j3482624-svc).

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:154]: failed to scale up node pool np-update-NODES from 2 to 3 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc001f44210>: 
    failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-7pwjpcnwnctq" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-NODES in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-7pwjpcnwnctq, error: context deadline exceeded
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_update_nodes.go:154]: failed to scale up node pool np-update-NODES from 2 to 3 replicas
Unexpected error:
    <*fmt.wrapErrors | 0xc001f44210>: 
    failed waiting for nodepool="np-update-NODES" in cluster="np-update-nodes-hcp-cluster" resourcegroup="nodepool-update-nodes-7pwjpcnwnctq" to finish updating, caused by: timeout '20.000000' minutes exceeded during UpdateNodePoolAndWait for nodepool np-update-NODES in cluster np-update-nodes-hcp-cluster in resource group nodepool-update-nodes-7pwjpcnwnctq, error: context deadline exceeded
    ...
occurred

time=2026-09-23T01:45:47.799Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-23T01:45:48.393Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j0769280-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:westus3 role:hcp-underlay-ci01-j0769280-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:1190e8af-ea6a-4410-bc47-5facec337dc4 region:westus3 secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#1190e8af-ea6a-4410-bc47-5facec337dc4 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-23T01:45:48.393Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T01:45:48.667Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-23T01:45:49.209Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-23T01:45:49.209Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T01:45:49.334Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T01:55:54.330Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-23T01:55:54.331Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-23T01:56:54.333Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-23T01:56:55.421Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j0769280-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:westus3 role:hcp-underlay-ci01-j0769280-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:1190e8af-ea6a-4410-bc47-5facec337dc4 region:westus3 secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#1190e8af-ea6a-4410-bc47-5facec337dc4 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-23T01:56:55.421Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T01:56:55.777Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-23T01:56:56.324Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T01:56:56.324Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T01:56:56.630Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-23T01:56:57.690Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T01:56:57.691Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-23T01:56:58.299Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-23T01:56:58.862Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-23T01:56:59.423Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-23T01:57:00.012Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-23T01:57:00.364Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-23T01:57:00.910Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-23T01:57:01.301Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-23T01:57:01.301Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T01:57:01.546Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=1
time=2026-09-23T02:07:04.175Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-23T02:07:04.175Z level=INFO msg="Retrying step" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit step=arobit error="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/arobit/arobit-forwarder not ready. status: InProgress, message: Available: 0/4\ncontext deadline exceeded"
time=2026-09-23T02:08:04.175Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit description="Step arobit\n Kind: Helm\n"
time=2026-09-23T02:08:05.008Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit values="map[forwarder:map[azureCloudEnvironmentName:AzurePublicCloud clusterName:ci01-j0769280-svc clusterType:svc dynamicConfig:false environment:dev fluentbit:map[image:map[digest:sha256:9f61c8da15b7578f04d809e2ee60266aca19fc9aab53f382123dafe0fee6f818 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:oss/v2/fluent/fluent-bit] resources:map[limits:map[memory:1248Mi] requests:map[cpu:250m memory:256Mi]]] geoShortId:us kusto:map[blob_uri_length:128 buffer_dir:/var/kusto buffer_file_delete_early:false buffering:true buffering_enabled:true delete_on_max_upload_error:true enabled:true hostedControlPlaneLogsDatabase:HostedControlPlaneLogs ingestionEndpoint:https://ingest-hcp-dev-us-2.eastus2.kusto.windows.net io_timeout:60s scheduler_max_retries:9 serviceLogsDatabase:ServiceLogs store_dir_limit_size:1.8GB unify_tag:true upload_file_size:16M upload_timeout:1m] mdsd:map[enabled:false geneva:map[clusterLogsAccountName:placeholder clusterLogsNamespace: clusterLogsSan:empty-sentinel clusterLogsSecretName:clusterlogs configVersion:1 environment:Test region:westus3 role:hcp-underlay-ci01-j0769280-svc rpAccountName:placeholder rpNamespace: rpSan:empty-sentinel rpSecretName:rplogs] image:map[digest:sha256:5eca8f713c64383bc545dd55f4af4020fd5db0c392d891cdc441765a2f7e52c3 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:geneva/distroless/mdsd]] msiClientId:1190e8af-ea6a-4410-bc47-5facec337dc4 region:westus3 secretProvider:map[keyVault:aro-hcp-dev-svc-kv] securityContext:map[enabled:true runAsGroup:65532 runAsUser:65532] serviceAccountName:arobit-forwarder tenantId:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX watchdog:map[image:map[digest:sha256:1fa09488bf407f4b1bfd44f02ff60af191218075be0a2050d737165acfcbda23 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:aks/command/runtime] intervalSeconds:15 timeoutSeconds:300]] watson:map[env:map[AUTHENTICATION_ID:MsiClientId#1190e8af-ea6a-4410-bc47-5facec337dc4 INCOMING_QUEUE:/azure-watson/cores MDSD_ROLE_PREFIX:/var/run/mdsd/asa/ REPORT_QUEUE:/azure-watson/queue WA_CORE_PATTERN:/azure-watson/cores/%e.%p.%h.%t] image:map[digest:sha256:8e5ef3e76be33794cef7460712eb9ff0c73bce8b8e5ffc33c5c6bfe0762f5db0 pullPolicy:IfNotPresent registry:mcr.microsoft.com repository:azure-watson/agent/azlinux3] resources:map[limits:map[cpu:250m memory:512Mi] requests:map[cpu:50m memory:128Mi]] securityContext:map[privileged:true] volumes:map[crashdumpHostPath:/home/docker/azure-watson]]]"
time=2026-09-23T02:08:05.008Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T02:08:05.466Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit namespace=arobit
time=2026-09-23T02:08:06.204Z level=INFO msg="Doing dry-run of Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T02:08:06.204Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T02:08:06.382Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-23T02:08:07.458Z level=INFO msg="Removing old field manager from objects in release manifest." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T02:08:07.458Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ServiceAccount" namespace=arobit name=arobit-forwarder
time=2026-09-23T02:08:07.843Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=ConfigMap" namespace=arobit name=arobit-forwarder
time=2026-09-23T02:08:08.593Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRole" namespace="" name=arobit-forwarder
time=2026-09-23T02:08:09.447Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding" namespace="" name=arobit-forwarder
time=2026-09-23T02:08:09.853Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="/v1, Kind=Service" namespace=arobit name=arobit-forwarder
time=2026-09-23T02:08:10.521Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="apps/v1, Kind=DaemonSet" namespace=arobit name=arobit-forwarder
time=2026-09-23T02:08:11.194Z level=INFO msg="Decoded resource from manifests." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit gvk="monitoring.coreos.com/v1, Kind=ServiceMonitor" namespace=arobit name=arobit-forwarder
time=2026-09-23T02:08:11.889Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit dryRun=false
time=2026-09-23T02:08:11.889Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit
time=2026-09-23T02:08:12.134Z level=INFO msg="Found release history, upgrading..." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit numVersions=2
time=2026-09-23T02:15:47.989Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="context deadline exceeded"
time=2026-09-23T02:15:47.990Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=service step=arobit err="error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: context deadline exceeded"

alert fired 4 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T01:52:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0769280-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-ch8gg", region="westus3", severity="warning"
  Description: Pod arobit/arobit-forwarder-ch8gg has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-23T01:52:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0769280-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-9ptjq", region="westus3", severity="warning"
  Description: Pod arobit/arobit-forwarder-9ptjq has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Fired (not resolved)
  Started:     2026-09-23T01:52:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0769280-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-kw5rf", region="westus3", severity="warning"
  Description: Pod arobit/arobit-forwarder-kw5rf has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Fired (not resolved)
  Started:     2026-09-23T01:52:03Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j0769280-svc", component="kubernetes-infrastructure", namespace="arobit", pod="arobit-forwarder-cxgg8", region="westus3", severity="warning"
  Description: Pod arobit/arobit-forwarder-cxgg8 has been in a non-ready state for longer than 5 minutes.

alert fired 2 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T02:02:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0769280-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0769280/providers/microsoft.monitor/accounts/services-j0769280", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-wqtm5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.
Firing 2:
  State:       Fired (not resolved)
  Started:     2026-09-23T02:02:05Z
  Severity:    Sev3
  Labels:      alertname="KubeDaemonSetRolloutStuck", cluster="ci01-j0769280-svc", component="kubernetes-infrastructure", container="kube-state-metrics", daemonset="arobit-forwarder", endpoint="http", environment="ci01", instance="10.128.64.47:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j0769280/providers/microsoft.monitor/accounts/services-j0769280", namespace="arobit", pod="arohcp-monitor-kube-state-metrics-849cb69576-wqtm5", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="westus3", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: DaemonSet arobit/arobit-forwarder has not finished or progressed for at least 15 minutes.

alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-23T02:02:02Z
  Severity:    Sev3
  Labels:      alertname="ArobitForwarderJobUp", cluster="ci01-j0769280-svc", component="monitoring-infrastructure", region="westus3", severity="warning"
  Description: The Arobit forwarder metrics endpoint on cluster ci01-j0769280-svc has been unreachable for 15 minutes.

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:202]: cluster CA data must contain exactly one PEM block
Expected
    <[]uint8 | len:6010, cap:6010>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:202]: cluster CA data must contain exactly one PEM block
Expected
    <[]uint8 | len:6010, cap:6010>: ...
to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:202]: cluster CA data must contain exactly one PEM block
Expected
    <[]uint8 | len:6010, cap:6010>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:202]: cluster CA data must contain exactly one PEM block
Expected
    <[]uint8 | len:6010, cap:6010>: ...
to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc000bc2d80>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-rdvqqgkbcqkm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a8f31d96-4bf3-4b48-a764-fe35d5af46db
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a8f31d96-4bf3-4b48-a764-fe35d5af46db",
      "name": "a8f31d96-4bf3-4b48-a764-fe35d5af46db",
      "status": "Failed",
      "startTime": "2026-09-23T23:51:44.432225393Z",
      "endTime": "2026-09-24T00:10:47.655792485Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:141]: failed to create node pool np-1 with initial labels and taints
Unexpected error:
    <*fmt.wrapError | 0xc000bc2d80>: 
    failed waiting for nodepool="np-1" for cluster "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-rdvqqgkbcqkm" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a8f31d96-4bf3-4b48-a764-fe35d5af46db
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/a8f31d96-4bf3-4b48-a764-fe35d5af46db",
      "name": "a8f31d96-4bf3-4b48-a764-fe35d5af46db",
      "status": "Failed",
      "startTime": "2026-09-23T23:51:44.432225393Z",
      "endTime": "2026-09-24T00:10:47.655792485Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009fe4a0>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-9tsbffh9csll": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-9tsbffh9csll/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009fe4a0>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-9tsbffh9csll": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-9tsbffh9csll/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc000cca500>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-4fchkp" in resourcegroup="rg-np-skip-minor-4fchkp-d9wq5jxbr4vg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-4fchkp-d9wq5jxbr4vg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-4fchkp/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc000cca500>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-4fchkp" in resourcegroup="rg-np-skip-minor-4fchkp-d9wq5jxbr4vg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-4fchkp-d9wq5jxbr4vg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-4fchkp/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000f96700>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-nfm8pxmkddmn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-nfm8pxmkddmn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000f96700>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-nfm8pxmkddmn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-nfm8pxmkddmn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc000fd6f60>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-7225cv" in resourcegroup="rg-np-version-upgrade-7225cv-t79q2jkmqh7t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7225cv-t79q2jkmqh7t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-7225cv/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc000fd6f60>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-7225cv" in resourcegroup="rg-np-version-upgrade-7225cv-t79q2jkmqh7t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7225cv-t79q2jkmqh7t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-7225cv/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:91]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000587ac0>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-f9strbqhzqsp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-f9strbqhzqsp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:91]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000587ac0>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-f9strbqhzqsp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-f9strbqhzqsp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0013365e0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-87rcp5g2p96w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-87rcp5g2p96w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0013365e0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-87rcp5g2p96w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-87rcp5g2p96w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc002a7cd50>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-lqwwzcqjkl4d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc002a7cd50>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-lqwwzcqjkl4d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc0010f5220>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-gp6h5d2jnk5d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-gp6h5d2jnk5d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc0010f5220>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-gp6h5d2jnk5d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-gp6h5d2jnk5d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-lqb7d8" for cluster "cluster-nightly-4-20-lqb7d8"
Unexpected error:
    <*fmt.wrapError | 0xc001181280>: 
    failed to create NodePool np-lqb7d8: failed starting nodepool creation "np-lqb7d8" for cluster "cluster-nightly-4-20-lqb7d8" in resourcegroup="rg-nightly-4-20-9g5dwcgvzm8b": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-9g5dwcgvzm8b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-lqb7d8/nodePools/np-lqb7d8
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:176]: failed to create node pool "np-lqb7d8" for cluster "cluster-nightly-4-20-lqb7d8"
Unexpected error:
    <*fmt.wrapError | 0xc001181280>: 
    failed to create NodePool np-lqb7d8: failed starting nodepool creation "np-lqb7d8" for cluster "cluster-nightly-4-20-lqb7d8" in resourcegroup="rg-nightly-4-20-9g5dwcgvzm8b": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-9g5dwcgvzm8b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-lqb7d8/nodePools/np-lqb7d8
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000618ec0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-b5n2hfst68hj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-b5n2hfst68hj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000618ec0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-b5n2hfst68hj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-b5n2hfst68hj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc001570180>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-9vczdt9qztzh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc001570180>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-9vczdt9qztzh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc002564810>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-cr6zxd6vcc7j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:275]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc002564810>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-cr6zxd6vcc7j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc00104b000>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-4t6q2594ttzv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-4t6q2594ttzv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc00104b000>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-4t6q2594ttzv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-4t6q2594ttzv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-mx2sdx5hq5cq/cluster-nightly-5-1-r4tn78 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001788aa0>: 
    failed to create HCP cluster cluster-nightly-5-1-r4tn78: failed waiting for cluster="cluster-nightly-5-1-r4tn78" in resourcegroup="rg-nightly-5-1-mx2sdx5hq5cq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9e1c75fb-14a3-4cb5-b5d0-9a40f34a6079
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9e1c75fb-14a3-4cb5-b5d0-9a40f34a6079",
      "name": "9e1c75fb-14a3-4cb5-b5d0-9a40f34a6079",
      "status": "Failed",
      "startTime": "2026-09-23T23:28:28.748187971Z",
      "endTime": "2026-09-23T23:57:28.896085734Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-mx2sdx5hq5cq/cluster-nightly-5-1-r4tn78 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001788aa0>: 
    failed to create HCP cluster cluster-nightly-5-1-r4tn78: failed waiting for cluster="cluster-nightly-5-1-r4tn78" in resourcegroup="rg-nightly-5-1-mx2sdx5hq5cq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9e1c75fb-14a3-4cb5-b5d0-9a40f34a6079
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/9e1c75fb-14a3-4cb5-b5d0-9a40f34a6079",
      "name": "9e1c75fb-14a3-4cb5-b5d0-9a40f34a6079",
      "status": "Failed",
      "startTime": "2026-09-23T23:28:28.748187971Z",
      "endTime": "2026-09-23T23:57:28.896085734Z",
      "error": {
        "code": "InternalServerError",
        "message": "[cosmosCluster] .api.url is empty; [servingCABundle] ServingCABundle not yet populated; [clusterServiceClusterStatus] cluster service has not been successfully created; [hypershiftHostedCluster] hosted cluster state not cached yet"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc001010900>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-2lsdzccng668/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc001010900>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-2lsdzccng668/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0013ce030>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-dcrp5phx2tdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0013ce030>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-dcrp5phx2tdt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc00121ed00>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-zj2nbd" resourcegroup="rg-np-dg-minor-zj2nbd-nbv87cpm7mgj": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-zj2nbd-nbv87cpm7mgj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-zj2nbd/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc00121ed00>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-zj2nbd" resourcegroup="rg-np-dg-minor-zj2nbd-nbv87cpm7mgj": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-zj2nbd-nbv87cpm7mgj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-zj2nbd/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-23-090319\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc000cbd410>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-23-031212, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.001s.
Expected success, but got an error:
    <*errors.joinError | 0xc000cbd410>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-23-031212, targetMinor=5.1.0-0.nightly-multi-2026-08-10-113053) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-08-10-113053"
    ...

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000c08920>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-xlkqvsjbnblh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-xlkqvsjbnblh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000c08920>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-xlkqvsjbnblh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-xlkqvsjbnblh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e91540>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-6r7w984km7pq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-6r7w984km7pq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e91540>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-6r7w984km7pq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-6r7w984km7pq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000e9ede0>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-zs28fdvptj4b": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-zs28fdvptj4b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000e9ede0>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-zs28fdvptj4b": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-zs28fdvptj4b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc001013f60>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-hn98pgcl2759": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-hn98pgcl2759/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc001013f60>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-hn98pgcl2759": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-hn98pgcl2759/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:202]: cluster CA data must contain exactly one PEM block
Expected
    <[]uint8 | len:6010, cap:6012>: ...
to be empty
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:202]: cluster CA data must contain exactly one PEM block
Expected
    <[]uint8 | len:6010, cap:6012>: ...
to be empty

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc000fd38c0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-d8vr2mxrtk2q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc000fd38c0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-d8vr2mxrtk2q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000a27d40>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-6t9mhgvsbfj6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-6t9mhgvsbfj6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000a27d40>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-6t9mhgvsbfj6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-6t9mhgvsbfj6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000bfede0>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-9ssdcjgnxb8x": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-9ssdcjgnxb8x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000bfede0>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-9ssdcjgnxb8x": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-9ssdcjgnxb8x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Required value: field cannot be null",
        "target": "properties.platform.vnetIntegrationSubnetId"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc002b8f878>: 
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-v52dlnd99fqc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8dff53d4-fb68-4baa-8655-9059740a3aee
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8dff53d4-fb68-4baa-8655-9059740a3aee",
      "name": "8dff53d4-fb68-4baa-8655-9059740a3aee",
      "status": "Failed",
      "startTime": "2026-09-23T02:19:20.475025137Z",
      "endTime": "2026-09-23T02:38:20.795241715Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:404]: failed to verify cilium is running and nodes are Ready for cluster "agg-cluster"
Unexpected error:
    <*errors.joinError | 0xc002b8f878>: 
    failed to create NodePool agg-np-a: failed waiting for nodepool="agg-np-a" for cluster "agg-cluster" in resourcegroup="feature-aggregation-v52dlnd99fqc" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8dff53d4-fb68-4baa-8655-9059740a3aee
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/8dff53d4-fb68-4baa-8655-9059740a3aee",
      "name": "8dff53d4-fb68-4baa-8655-9059740a3aee",
      "status": "Failed",
      "startTime": "2026-09-23T02:19:20.475025137Z",
      "endTime": "2026-09-23T02:38:20.795241715Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:82]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc0012078e0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-np-labels-taints-lsql8k" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-np-labels-taints-lsql8k/operationStatuses/08584113986461824979
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-9lbzntwzvchs/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-r6cuakilyx67k\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-14/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:82]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc0012078e0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-np-labels-taints-lsql8k" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-np-labels-taints-lsql8k/operationStatuses/08584113986461824979
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-9lbzntwzvchs/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-r6cuakilyx67k\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-14/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc0006ead40>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-g6c2l8pqzc65": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-g6c2l8pqzc65/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc0006ead40>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-g6c2l8pqzc65": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-g6c2l8pqzc65/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-b77rqs" with version 4.22.14 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b2a3c0>: 
    failed to create HCP cluster cluster-zstream-4-22-b77rqs: failed starting cluster creation "cluster-zstream-4-22-b77rqs" in resourcegroup="rg-zstream-upgrade-4-22-x86pcxvscv78": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-x86pcxvscv78/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-b77rqs
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-b77rqs" with version 4.22.14 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b2a3c0>: 
    failed to create HCP cluster cluster-zstream-4-22-b77rqs: failed starting cluster creation "cluster-zstream-4-22-b77rqs" in resourcegroup="rg-zstream-upgrade-4-22-x86pcxvscv78": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-x86pcxvscv78/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-b77rqs
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc000f1a280>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-zsl2zqg8d25s": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-zsl2zqg8d25s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc000f1a280>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-zsl2zqg8d25s": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-zsl2zqg8d25s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000fa20e0>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-ng56lz7dl9cb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ng56lz7dl9cb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000fa20e0>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-ng56lz7dl9cb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-ng56lz7dl9cb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000f8c120>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-qx726fw4j5rl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-qx726fw4j5rl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000f8c120>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-qx726fw4j5rl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-qx726fw4j5rl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-wc8jdx" with version 4.20.38 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0009e0640>: 
    failed to create HCP cluster cluster-zstream-4-20-wc8jdx: failed starting cluster creation "cluster-zstream-4-20-wc8jdx" in resourcegroup="rg-zstream-upgrade-4-20-9dr7b9qsqclg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-9dr7b9qsqclg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-wc8jdx
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-wc8jdx" with version 4.20.38 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0009e0640>: 
    failed to create HCP cluster cluster-zstream-4-20-wc8jdx: failed starting cluster creation "cluster-zstream-4-20-wc8jdx" in resourcegroup="rg-zstream-upgrade-4-20-9dr7b9qsqclg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-9dr7b9qsqclg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-wc8jdx
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:78]: failed to create customer resources for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000a092e0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ea-listxdjmk6-rp7t27" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-listxdjmk6-rp7t27/operationStatuses/08584113987138459688
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-g8gfj6vr8skb/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-rlufqgosrjhbk\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-07/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:78]: failed to create customer resources for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000a092e0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ea-listxdjmk6-rp7t27" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-listxdjmk6-rp7t27/operationStatuses/08584113987138459688
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-g8gfj6vr8skb/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-rlufqgosrjhbk\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-07/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:283]: failed to create cluster customer resources for node-limit test
Unexpected error:
    <*fmt.wrapError | 0xc000619cc0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-fzl4fk" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-fzl4fk/operationStatuses/08584113987139523817
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-23/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-brhj6q4dslz6/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:283]: failed to create cluster customer resources for node-limit test
Unexpected error:
    <*fmt.wrapError | 0xc000619cc0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-node-limit-cluster-fzl4fk" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-node-limit-cluster-fzl4fk/operationStatuses/08584113987139523817
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-23/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-brhj6q4dslz6/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-j6nw3ohx7bzo4\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-97lxrpzvwp8t/cluster-candidate-4-21-mq5ksz should provision
Unexpected error:
    <*fmt.wrapError | 0xc000982100>: 
    failed to create HCP cluster cluster-candidate-4-21-mq5ksz: failed starting cluster creation "cluster-candidate-4-21-mq5ksz" in resourcegroup="rg-candidate-4-21-97lxrpzvwp8t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-97lxrpzvwp8t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-21-mq5ksz
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-97lxrpzvwp8t/cluster-candidate-4-21-mq5ksz should provision
Unexpected error:
    <*fmt.wrapError | 0xc000982100>: 
    failed to create HCP cluster cluster-candidate-4-21-mq5ksz: failed starting cluster creation "cluster-candidate-4-21-mq5ksz" in resourcegroup="rg-candidate-4-21-97lxrpzvwp8t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-97lxrpzvwp8t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-21-mq5ksz
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc0007d6fa0>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-n662f9vjlzcr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-n662f9vjlzcr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc0007d6fa0>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-n662f9vjlzcr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-n662f9vjlzcr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:136]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc0001e4b80>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-42bbcr-vkf8rz" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-42bbcr-vkf8rz/operationStatuses/08584113986227757596
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-42bbcr-2rk6t6mlf2lm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-pgend26hbad42\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-02/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:136]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc0001e4b80>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-42bbcr-vkf8rz" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-42bbcr-vkf8rz/operationStatuses/08584113986227757596
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-42bbcr-2rk6t6mlf2lm/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-pgend26hbad42\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-02/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-hlvkx99zwmlc/cluster-candidate-4-22-wjv59w should provision
Unexpected error:
    <*fmt.wrapError | 0xc00101a580>: 
    failed to create HCP cluster cluster-candidate-4-22-wjv59w: failed starting cluster creation "cluster-candidate-4-22-wjv59w" in resourcegroup="rg-candidate-4-22-hlvkx99zwmlc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-hlvkx99zwmlc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-22-wjv59w
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-hlvkx99zwmlc/cluster-candidate-4-22-wjv59w should provision
Unexpected error:
    <*fmt.wrapError | 0xc00101a580>: 
    failed to create HCP cluster cluster-candidate-4-22-wjv59w: failed starting cluster creation "cluster-candidate-4-22-wjv59w" in resourcegroup="rg-candidate-4-22-hlvkx99zwmlc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-hlvkx99zwmlc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-22-wjv59w
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000c94ac0>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-gtn8glz92n2v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-gtn8glz92n2v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000c94ac0>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-gtn8glz92n2v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-gtn8glz92n2v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-sjhtzvzlt9l4/cluster-candidate-4-23-5ld87l should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f02240>: 
    failed to create HCP cluster cluster-candidate-4-23-5ld87l: failed starting cluster creation "cluster-candidate-4-23-5ld87l" in resourcegroup="rg-candidate-4-23-sjhtzvzlt9l4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sjhtzvzlt9l4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-23-5ld87l
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-23-sjhtzvzlt9l4/cluster-candidate-4-23-5ld87l should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f02240>: 
    failed to create HCP cluster cluster-candidate-4-23-5ld87l: failed starting cluster creation "cluster-candidate-4-23-5ld87l" in resourcegroup="rg-candidate-4-23-sjhtzvzlt9l4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-sjhtzvzlt9l4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-23-5ld87l
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:85]: failed to create customer resources for external auth cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c1e3a0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ea-cluster-66dgzj" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-cluster-66dgzj/operationStatuses/08584113987138933581
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-kvx8wtp49bjz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ltdphft7fgrwg\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-03/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:85]: failed to create customer resources for external auth cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c1e3a0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ea-cluster-66dgzj" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ea-cluster-66dgzj/operationStatuses/08584113987138933581
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-kvx8wtp49bjz/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ltdphft7fgrwg\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-03/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000d80380>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-fxmxrgw75sbn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-fxmxrgw75sbn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000d80380>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-fxmxrgw75sbn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-fxmxrgw75sbn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000c825a0>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-xxnn2gk7lrb2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-xxnn2gk7lrb2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000c825a0>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-xxnn2gk7lrb2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-xxnn2gk7lrb2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:106]: failed to create customer resources for z-stream cluster "cluster-zstream-4-21-xcgpz6"
Unexpected error:
    <*fmt.wrapError | 0xc000f00c80>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-21-xcgpz6-4zrvmh" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-xcgpz6-4zrvmh/operationStatuses/08584113987705963996
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-02/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:106]: failed to create customer resources for z-stream cluster "cluster-zstream-4-21-xcgpz6"
Unexpected error:
    <*fmt.wrapError | 0xc000f00c80>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cluster-zstream-4-21-xcgpz6-4zrvmh" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cluster-zstream-4-21-xcgpz6-4zrvmh/operationStatuses/08584113987705963996
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-02/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000712140>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-cl52xgnhc4wx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-cl52xgnhc4wx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000712140>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-cl52xgnhc4wx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-cl52xgnhc4wx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-dm79nndfl6r5/cluster-candidate-5-0-x7bgf9 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00184e0e0>: 
    failed to create HCP cluster cluster-candidate-5-0-x7bgf9: failed starting cluster creation "cluster-candidate-5-0-x7bgf9" in resourcegroup="rg-candidate-5-0-dm79nndfl6r5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-dm79nndfl6r5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-x7bgf9
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-dm79nndfl6r5/cluster-candidate-5-0-x7bgf9 should provision
Unexpected error:
    <*fmt.wrapError | 0xc00184e0e0>: 
    failed to create HCP cluster cluster-candidate-5-0-x7bgf9: failed starting cluster creation "cluster-candidate-5-0-x7bgf9" in resourcegroup="rg-candidate-5-0-dm79nndfl6r5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-dm79nndfl6r5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-x7bgf9
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:123]: failed to create customer resources for y-stream upgrade cluster "cp-ystream-upgrade-4-21-jrrftk"
Unexpected error:
    <*fmt.wrapError | 0xc000da6360>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cp-ystream-upgrade-4-21-jrrftk-g7vqn6" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cp-ystream-upgrade-4-21-jrrftk-g7vqn6/operationStatuses/08584113987434500608
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-l442rxkb7vdp/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-mpj57e7od7fwq\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-13/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:123]: failed to create customer resources for y-stream upgrade cluster "cp-ystream-upgrade-4-21-jrrftk"
Unexpected error:
    <*fmt.wrapError | 0xc000da6360>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cp-ystream-upgrade-4-21-jrrftk-g7vqn6" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cp-ystream-upgrade-4-21-jrrftk-g7vqn6/operationStatuses/08584113987434500608
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-l442rxkb7vdp/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-mpj57e7od7fwq\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-13/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:123]: failed to create customer resources for y-stream upgrade cluster "cp-ystream-upgrade-4-22-vvgtcx"
Unexpected error:
    <*fmt.wrapError | 0xc000d7cb00>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cp-ystream-upgrade-4-22-vvgtcx-tmrhxj" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cp-ystream-upgrade-4-22-vvgtcx-tmrhxj/operationStatuses/08584113987427336698
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-18/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-dpmbrwp9fml7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-mem35qo2kwqbg\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:123]: failed to create customer resources for y-stream upgrade cluster "cp-ystream-upgrade-4-22-vvgtcx"
Unexpected error:
    <*fmt.wrapError | 0xc000d7cb00>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cp-ystream-upgrade-4-22-vvgtcx-tmrhxj" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cp-ystream-upgrade-4-22-vvgtcx-tmrhxj/operationStatuses/08584113987427336698
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-18/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-dpmbrwp9fml7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-mem35qo2kwqbg\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:136]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc001120500>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-swjhtx-jx27v7" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-swjhtx-jx27v7/operationStatuses/08584113986224996378
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-04/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-swjhtx-nll5kh9rqz25/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-sfpatc6wjwnlg\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:136]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc001120500>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-version-upgrade-cluster-swjhtx-jx27v7" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-version-upgrade-cluster-swjhtx-jx27v7/operationStatuses/08584113986224996378
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-04/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-swjhtx-nll5kh9rqz25/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-sfpatc6wjwnlg\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:91]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000fec1e0>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-t4c48wqpwt72": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-t4c48wqpwt72/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:91]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000fec1e0>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-t4c48wqpwt72": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-t4c48wqpwt72/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:101]: failed to begin creating HCP cluster "admin-cred-lifecycle-v7kjhv"
Unexpected error:
    <*fmt.wrapError | 0xc000d905c0>: 
    failed starting cluster creation "admin-cred-lifecycle-v7kjhv" in resourcegroup="admin-credential-lifecycle-test-qws7j9zzgwzk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-qws7j9zzgwzk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-v7kjhv
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:101]: failed to begin creating HCP cluster "admin-cred-lifecycle-v7kjhv"
Unexpected error:
    <*fmt.wrapError | 0xc000d905c0>: 
    failed starting cluster creation "admin-cred-lifecycle-v7kjhv" in resourcegroup="admin-credential-lifecycle-test-qws7j9zzgwzk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-qws7j9zzgwzk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-v7kjhv
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:90]: failed to create customer resources for IDMS cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f8ef20>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-idms-e2e-hcp-cluster-f4gj82" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-idms-e2e-hcp-cluster-f4gj82/operationStatuses/08584113987063020089
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-08/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-j8rcf7j8sc84/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-lv7dvzspd2e5i\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:90]: failed to create customer resources for IDMS cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f8ef20>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-idms-e2e-hcp-cluster-f4gj82" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-idms-e2e-hcp-cluster-f4gj82/operationStatuses/08584113987063020089
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-08/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-j8rcf7j8sc84/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-lv7dvzspd2e5i\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:291]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc00101b980>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-noedge-6qq489-c5jsdg" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-noedge-6qq489-c5jsdg/operationStatuses/08584113985901622419
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-19/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-6qq489-tpss6kb9nfnv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-yslh3n7rfjni4\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:291]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc00101b980>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-noedge-6qq489-c5jsdg" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-noedge-6qq489-c5jsdg/operationStatuses/08584113985901622419
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-19/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-6qq489-tpss6kb9nfnv/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-yslh3n7rfjni4\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c9a2e0>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-pptmdm5h47tr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-pptmdm5h47tr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c9a2e0>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-pptmdm5h47tr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-pptmdm5h47tr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc000e923f0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-z5x8hr8lv45s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc000e923f0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-z5x8hr8lv45s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:79]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc0007876a0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-disabled-image-registry-hcp-cluster-lrqgv6" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-lrqgv6/operationStatuses/08584113987055376578
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-09/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-xh4nq445fjw9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-qmh74zogaodhg\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:79]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc0007876a0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-disabled-image-registry-hcp-cluster-lrqgv6" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-disabled-image-registry-hcp-cluster-lrqgv6/operationStatuses/08584113987055376578
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-09/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-xh4nq445fjw9/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-qmh74zogaodhg\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:80]: failed to create cluster customer resources for cordon test
Unexpected error:
    <*fmt.wrapError | 0xc00032e320>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cordon-hcp-cluster-rnbmsq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cordon-hcp-cluster-rnbmsq/operationStatuses/08584113987137499225
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-24/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-wcgtvsqfsxxx/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3wbuyr3ac6kzo\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:80]: failed to create cluster customer resources for cordon test
Unexpected error:
    <*fmt.wrapError | 0xc00032e320>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cordon-hcp-cluster-rnbmsq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cordon-hcp-cluster-rnbmsq/operationStatuses/08584113987137499225
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-24/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-wcgtvsqfsxxx/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-3wbuyr3ac6kzo\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc00180e180>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-47vf99jdbhj4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-47vf99jdbhj4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc00180e180>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-47vf99jdbhj4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-47vf99jdbhj4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc00051f700>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-mf589sk4rj9r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-mf589sk4rj9r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc00051f700>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-mf589sk4rj9r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-mf589sk4rj9r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:570]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc001026280>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-downgrade-shn7fn-zvxvfc" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-downgrade-shn7fn-zvxvfc/operationStatuses/08584113985869276090
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-shn7fn-psrg7w9f9xgl/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ev3yo2onirsd6\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:570]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc001026280>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-downgrade-shn7fn-zvxvfc" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-downgrade-shn7fn-zvxvfc/operationStatuses/08584113985869276090
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-shn7fn-psrg7w9f9xgl/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ev3yo2onirsd6\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000c18180>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-bwh9fhw68vj7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-bwh9fhw68vj7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000c18180>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-bwh9fhw68vj7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-bwh9fhw68vj7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e04180>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-zwxh6x2kssck": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-zwxh6x2kssck/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e04180>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-zwxh6x2kssck": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-zwxh6x2kssck/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0005d3560>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-d7mvztslkt8z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-d7mvztslkt8z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0005d3560>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-d7mvztslkt8z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-d7mvztslkt8z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc001092280>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-f5hpwzxskrlb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-f5hpwzxskrlb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc001092280>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-f5hpwzxskrlb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-f5hpwzxskrlb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:268]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000c1cc00>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-shoebox-hcp-cluster-w9k5pm" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-shoebox-hcp-cluster-w9k5pm/operationStatuses/08584113985685912165
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-v8ln66whgcfk/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-22eudp3sazqeq\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-08/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:268]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000c1cc00>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-shoebox-hcp-cluster-w9k5pm" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-shoebox-hcp-cluster-w9k5pm/operationStatuses/08584113985685912165
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-v8ln66whgcfk/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-22eudp3sazqeq\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-08/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc000e81a40>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc000e81a40>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc00098eaa0>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-9fk2x7xsncqp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-9fk2x7xsncqp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc00098eaa0>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-9fk2x7xsncqp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-9fk2x7xsncqp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:83]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000bfaac0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-negative-tests-cluster-t75xkk" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-t75xkk/operationStatuses/08584113985598746400
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-xrwtbrzrms4p/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-03/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:83]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000bfaac0>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-negative-tests-cluster-t75xkk" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-negative-tests-cluster-t75xkk/operationStatuses/08584113985598746400
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-xrwtbrzrms4p/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-oeayaalhj7wjy\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-03/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000984920>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-zmq2fmksvd6w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-zmq2fmksvd6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000984920>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-zmq2fmksvd6w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-zmq2fmksvd6w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:93]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000c0ae00>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-autoscale-cluster-djz46v" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-djz46v/operationStatuses/08584113987138680420
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-wpsbhgpdgqb8/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-22/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:93]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000c0ae00>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-autoscale-cluster-djz46v" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-autoscale-cluster-djz46v/operationStatuses/08584113987138680420
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-wpsbhgpdgqb8/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-ssrwqrnhyyabs\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-22/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:75]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000ba8c00>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ephemeral-disk-5x6zpb" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ephemeral-disk-5x6zpb/operationStatuses/08584113986762247712
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-cplcmv2sq2w7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-6xts4ycasve7q\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-06/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:75]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000ba8c00>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-ephemeral-disk-5x6zpb" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-ephemeral-disk-5x6zpb/operationStatuses/08584113986762247712
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-cplcmv2sq2w7/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-6xts4ycasve7q\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-06/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:123]: failed to create customer resources for y-stream upgrade cluster "cp-ystream-upgrade-5-0-nmd797"
Unexpected error:
    <*fmt.wrapError | 0xc001752920>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cp-ystream-upgrade-5-0-nmd797-bjjz9t" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cp-ystream-upgrade-5-0-nmd797-bjjz9t/operationStatuses/08584113987415585751
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-20/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-qtbdqjphjldj/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-xfkchgv3ef7bk\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:123]: failed to create customer resources for y-stream upgrade cluster "cp-ystream-upgrade-5-0-nmd797"
Unexpected error:
    <*fmt.wrapError | 0xc001752920>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cp-ystream-upgrade-5-0-nmd797-bjjz9t" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cp-ystream-upgrade-5-0-nmd797-bjjz9t/operationStatuses/08584113987415585751
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-20/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-qtbdqjphjldj/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-xfkchgv3ef7bk\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-45rstd with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000902360>: 
    failed to create HCP cluster np-version-upgrade-cluster-45rstd: failed starting cluster creation "np-version-upgrade-cluster-45rstd" in resourcegroup="rg-np-version-upgrade-45rstd-nh6m8src8lfr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-45rstd-nh6m8src8lfr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-45rstd
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-45rstd with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc000902360>: 
    failed to create HCP cluster np-version-upgrade-cluster-45rstd: failed starting cluster creation "np-version-upgrade-cluster-45rstd" in resourcegroup="rg-np-version-upgrade-45rstd-nh6m8src8lfr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-45rstd-nh6m8src8lfr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-45rstd
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc0012820e0>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-nbwkpxzhms7s": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-nbwkpxzhms7s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc0012820e0>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-nbwkpxzhms7s": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-nbwkpxzhms7s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-ntq4zw" with version 5.0.0-rc.2 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0011ce260>: 
    failed to create HCP cluster cluster-zstream-5-0-ntq4zw: failed starting cluster creation "cluster-zstream-5-0-ntq4zw" in resourcegroup="rg-zstream-upgrade-5-0-hqk75qtw9hlt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-hqk75qtw9hlt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-ntq4zw
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-ntq4zw" with version 5.0.0-rc.2 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0011ce260>: 
    failed to create HCP cluster cluster-zstream-5-0-ntq4zw: failed starting cluster creation "cluster-zstream-5-0-ntq4zw" in resourcegroup="rg-zstream-upgrade-5-0-hqk75qtw9hlt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-hqk75qtw9hlt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-ntq4zw
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:218]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000b04c80>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-oidc-wi-cluster-2jwn4b" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-oidc-wi-cluster-2jwn4b/operationStatuses/08584113985700308617
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-23/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-rczl9wf28lwt/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-s6kwp7xg43dbi\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:218]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000b04c80>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-oidc-wi-cluster-2jwn4b" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-oidc-wi-cluster-2jwn4b/operationStatuses/08584113985700308617
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-23/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-rczl9wf28lwt/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-s6kwp7xg43dbi\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000d10500>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-l44lntsvmpb8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-l44lntsvmpb8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000d10500>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-l44lntsvmpb8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-l44lntsvmpb8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc00111e320>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-4bqqhnwdh8r5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-4bqqhnwdh8r5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc00111e320>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-4bqqhnwdh8r5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-4bqqhnwdh8r5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:431]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000408980>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-skip-minor-z8pbtk-44k8bq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-skip-minor-z8pbtk-44k8bq/operationStatuses/08584113985924298427
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-z8pbtk-p9lrjqz5xlmf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-a44nzwnsonewu\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-18/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:431]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc000408980>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-skip-minor-z8pbtk-44k8bq" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-skip-minor-z8pbtk-44k8bq/operationStatuses/08584113985924298427
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-z8pbtk-p9lrjqz5xlmf/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-a44nzwnsonewu\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-18/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:71]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc001180e80>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-delete-hcp-cluster-rdm6f7" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-delete-hcp-cluster-rdm6f7/operationStatuses/08584113987041544035
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-05/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-pxh9tsgnpxwk/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-rcpnvqrcxdqmo\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:71]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc001180e80>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-delete-hcp-cluster-rdm6f7" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-delete-hcp-cluster-rdm6f7/operationStatuses/08584113987041544035
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-05/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-pxh9tsgnpxwk/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-rcpnvqrcxdqmo\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-6sv9qtzkp7p5/cluster-candidate-4-20-k9rjvx should provision
Unexpected error:
    <*fmt.wrapError | 0xc000d46cc0>: 
    failed to create HCP cluster cluster-candidate-4-20-k9rjvx: failed starting cluster creation "cluster-candidate-4-20-k9rjvx" in resourcegroup="rg-candidate-4-20-6sv9qtzkp7p5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-6sv9qtzkp7p5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-20-k9rjvx
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-6sv9qtzkp7p5/cluster-candidate-4-20-k9rjvx should provision
Unexpected error:
    <*fmt.wrapError | 0xc000d46cc0>: 
    failed to create HCP cluster cluster-candidate-4-20-k9rjvx: failed starting cluster creation "cluster-candidate-4-20-k9rjvx" in resourcegroup="rg-candidate-4-20-6sv9qtzkp7p5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-6sv9qtzkp7p5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-20-k9rjvx
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:710]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc0006d0900>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-dg-minor-z4tc6h-tk4t87" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-dg-minor-z4tc6h-tk4t87/operationStatuses/08584113985739549250
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-24/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-z4tc6h-rxbv2s7dz2bg/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-h2mxcxvipvybg\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:710]: failed to create cluster customer resources
Unexpected error:
    <*fmt.wrapError | 0xc0006d0900>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-np-dg-minor-z4tc6h-tk4t87" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-np-dg-minor-z4tc6h-tk4t87/operationStatuses/08584113985739549250
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-24/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-z4tc6h-rxbv2s7dz2bg/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-h2mxcxvipvybg\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:123]: failed to create customer resources for y-stream upgrade cluster "cp-ystream-upgrade-4-23-t78srk"
Unexpected error:
    <*fmt.wrapError | 0xc0015fbe20>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cp-ystream-upgrade-4-23-t78srk-ftnpvj" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cp-ystream-upgrade-4-23-t78srk-ftnpvj/operationStatuses/08584113987507742946
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-19/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-v9krnbjlzvgl/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-gdp5fqxbc7uf4\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:123]: failed to create customer resources for y-stream upgrade cluster "cp-ystream-upgrade-4-23-t78srk"
Unexpected error:
    <*fmt.wrapError | 0xc0015fbe20>: 
    failed to create managed identities: failed to create managed identities: failed waiting for deployment "mi-cp-ystream-upgrade-4-23-t78srk-ftnpvj" at subscription scope to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/mi-cp-ystream-upgrade-4-23-t78srk-ftnpvj/operationStatuses/08584113987507742946
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-msi-container-stg-shard0-00-19/providers/Microsoft.Resources/deployments/pooledMsiScopedAssignments\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          },
          {
            "code": "Conflict",
            "message": "{\r\n  \"status\": \"Failed\",\r\n  \"error\": {\r\n    \"code\": \"ResourceDeploymentFailure\",\r\n    \"message\": \"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\",\r\n    \"details\": [\r\n      {\r\n        \"code\": \"DeploymentFailed\",\r\n        \"target\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-v9krnbjlzvgl/providers/Microsoft.Resources/deployments/pooledNonMsiScopedAssignments-gdp5fqxbc7uf4\",\r\n        \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\r\n        \"details\": [\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          },\r\n          {\r\n            \"code\": \"RoleAssignmentLimitExceeded\",\r\n            \"message\": \"No more role assignments can be created.\"\r\n          }\r\n        ]\r\n      }\r\n    ]\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00039efc0>: 
    failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-t9ljlvhpqrl2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-t9ljlvhpqrl2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00039efc0>: 
    failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-t9ljlvhpqrl2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-t9ljlvhpqrl2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000f4a300>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-wk479hrrr6dd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-wk479hrrr6dd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc000f4a300>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-wk479hrrr6dd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-wk479hrrr6dd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000a0c280>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-n2mwb88rc2kc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-n2mwb88rc2kc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000a0c280>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-n2mwb88rc2kc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-n2mwb88rc2kc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0004d5960>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-z4z66hnzq25r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-z4z66hnzq25r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc0004d5960>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-z4z66hnzq25r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-z4z66hnzq25r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc0015bb5f0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc0015bb5f0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-gvq7bc" at install version 4.22.0-0.nightly-multi-2026-09-23-151735
Unexpected error:
    <*fmt.wrapError | 0xc000c2e580>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-gvq7bc: failed starting cluster creation "cp-ystream-upgrade-5-0-gvq7bc" in resourcegroup="rg-cp-ystream-upgrade-5-0-5gq89l7l9zj4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-5gq89l7l9zj4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-gvq7bc
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-gvq7bc" at install version 4.22.0-0.nightly-multi-2026-09-23-151735
Unexpected error:
    <*fmt.wrapError | 0xc000c2e580>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-gvq7bc: failed starting cluster creation "cp-ystream-upgrade-5-0-gvq7bc" in resourcegroup="rg-cp-ystream-upgrade-5-0-5gq89l7l9zj4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-5gq89l7l9zj4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-gvq7bc
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000b582e0>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-blllqrwdhwg2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-blllqrwdhwg2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000b582e0>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-blllqrwdhwg2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-blllqrwdhwg2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f10140>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-jrntrv2b4mzw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-jrntrv2b4mzw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f10140>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-jrntrv2b4mzw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-jrntrv2b4mzw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000f12180>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-j75pdr9hg97t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-j75pdr9hg97t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000f12180>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-j75pdr9hg97t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-j75pdr9hg97t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc001072140>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-5v7nmx4glfms": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-5v7nmx4glfms/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc001072140>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-5v7nmx4glfms": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-5v7nmx4glfms/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000f14640>: 
    failed to create HCP cluster ea-list9flw4f: failed starting cluster creation "ea-list9flw4f" in resourcegroup="ea-list-89wl2cn26v96": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-89wl2cn26v96/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-list9flw4f
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000f14640>: 
    failed to create HCP cluster ea-list9flw4f: failed starting cluster creation "ea-list9flw4f" in resourcegroup="ea-list-89wl2cn26v96": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-89wl2cn26v96/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-list9flw4f
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-rckglw" at install version 4.21.0-0.nightly-multi-2026-09-23-170748
Unexpected error:
    <*fmt.wrapError | 0xc00069a7a0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-rckglw: failed starting cluster creation "cp-ystream-upgrade-4-22-rckglw" in resourcegroup="rg-cp-ystream-upgrade-4-22-7mwhqknsl2vx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-7mwhqknsl2vx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-rckglw
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-rckglw" at install version 4.21.0-0.nightly-multi-2026-09-23-170748
Unexpected error:
    <*fmt.wrapError | 0xc00069a7a0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-rckglw: failed starting cluster creation "cp-ystream-upgrade-4-22-rckglw" in resourcegroup="rg-cp-ystream-upgrade-4-22-7mwhqknsl2vx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-7mwhqknsl2vx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-rckglw
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000f166c0>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-fzxrb9z9hsxg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-fzxrb9z9hsxg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000f166c0>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-fzxrb9z9hsxg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-fzxrb9z9hsxg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-92b6l7" at install version 5.0.0-0.nightly-multi-2026-09-23-031212
Unexpected error:
    <*fmt.wrapError | 0xc000e642c0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-92b6l7: failed starting cluster creation "cp-ystream-upgrade-5-1-92b6l7" in resourcegroup="rg-cp-ystream-upgrade-5-1-8pm7wghtpk6v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-8pm7wghtpk6v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-92b6l7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-92b6l7" at install version 5.0.0-0.nightly-multi-2026-09-23-031212
Unexpected error:
    <*fmt.wrapError | 0xc000e642c0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-92b6l7: failed starting cluster creation "cp-ystream-upgrade-5-1-92b6l7" in resourcegroup="rg-cp-ystream-upgrade-5-1-8pm7wghtpk6v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-8pm7wghtpk6v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-92b6l7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-fqch49 at version 4.22.0-0.nightly-multi-2026-09-23-151735
Unexpected error:
    <*fmt.wrapError | 0xc000d9c380>: 
    failed to create HCP cluster np-skip-minor-fqch49: failed starting cluster creation "np-skip-minor-fqch49" in resourcegroup="rg-np-skip-minor-fqch49-rrvslkw6xp2h": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-fqch49-rrvslkw6xp2h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-fqch49
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-fqch49 at version 4.22.0-0.nightly-multi-2026-09-23-151735
Unexpected error:
    <*fmt.wrapError | 0xc000d9c380>: 
    failed to create HCP cluster np-skip-minor-fqch49: failed starting cluster creation "np-skip-minor-fqch49" in resourcegroup="rg-np-skip-minor-fqch49-rrvslkw6xp2h": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-fqch49-rrvslkw6xp2h/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-fqch49
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000b5a2a0>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-gfwfk6s8b9j2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-gfwfk6s8b9j2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000b5a2a0>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-gfwfk6s8b9j2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-gfwfk6s8b9j2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000912300>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-nmfgpbvxrwtl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-nmfgpbvxrwtl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000912300>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-nmfgpbvxrwtl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-nmfgpbvxrwtl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc0010d8200>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-fs6xcx4wqfx6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-fs6xcx4wqfx6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc0010d8200>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-fs6xcx4wqfx6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-fs6xcx4wqfx6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f16180>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-p4696z94nm5m": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-p4696z94nm5m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f16180>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-p4696z94nm5m": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-p4696z94nm5m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc001522260>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-97cn59jkhhl4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-97cn59jkhhl4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc001522260>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-97cn59jkhhl4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-97cn59jkhhl4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0008d7280>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-v6cldcx5jl9f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v6cldcx5jl9f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0008d7280>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-v6cldcx5jl9f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-v6cldcx5jl9f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc000fe2120>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-tq7lzxq6ld4m": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-tq7lzxq6ld4m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc000fe2120>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-tq7lzxq6ld4m": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-tq7lzxq6ld4m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f9c2a0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-xq29bq2p4r67": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-xq29bq2p4r67/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f9c2a0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-xq29bq2p4r67": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-xq29bq2p4r67/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-23-t2bxq8nv5246/cluster-nightly-4-23-p9fb96 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b28880>: 
    failed to create HCP cluster cluster-nightly-4-23-p9fb96: failed starting cluster creation "cluster-nightly-4-23-p9fb96" in resourcegroup="rg-nightly-4-23-t2bxq8nv5246": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-t2bxq8nv5246/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-23-p9fb96
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-23-t2bxq8nv5246/cluster-nightly-4-23-p9fb96 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000b28880>: 
    failed to create HCP cluster cluster-nightly-4-23-p9fb96: failed starting cluster creation "cluster-nightly-4-23-p9fb96" in resourcegroup="rg-nightly-4-23-t2bxq8nv5246": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-t2bxq8nv5246/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-23-p9fb96
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc0008843e0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-74cvpvwnrmg8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-74cvpvwnrmg8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc0008843e0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-74cvpvwnrmg8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-74cvpvwnrmg8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:91]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000d265e0>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-jwdzg46qwq7w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-jwdzg46qwq7w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:91]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000d265e0>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-jwdzg46qwq7w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-jwdzg46qwq7w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000457f20>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-dbdgjlmvwgtx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dbdgjlmvwgtx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000457f20>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-dbdgjlmvwgtx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-dbdgjlmvwgtx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-npwjgprz6msl/cluster-nightly-5-0-hfdrrw should provision
Unexpected error:
    <*fmt.wrapError | 0xc00108e140>: 
    failed to create HCP cluster cluster-nightly-5-0-hfdrrw: failed starting cluster creation "cluster-nightly-5-0-hfdrrw" in resourcegroup="rg-nightly-5-0-npwjgprz6msl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-npwjgprz6msl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-0-hfdrrw
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-0-npwjgprz6msl/cluster-nightly-5-0-hfdrrw should provision
Unexpected error:
    <*fmt.wrapError | 0xc00108e140>: 
    failed to create HCP cluster cluster-nightly-5-0-hfdrrw: failed starting cluster creation "cluster-nightly-5-0-hfdrrw" in resourcegroup="rg-nightly-5-0-npwjgprz6msl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-npwjgprz6msl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-0-hfdrrw
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000736180>: 
    failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-kszfsz4kjdp5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-kszfsz4kjdp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000736180>: 
    failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-kszfsz4kjdp5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-kszfsz4kjdp5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc0009c0160>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-khv8dvr7ks7m": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-khv8dvr7ks7m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc0009c0160>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-khv8dvr7ks7m": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-khv8dvr7ks7m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-9xw8wplswg6p/cluster-nightly-4-21-4bm8nq should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a941c0>: 
    failed to create HCP cluster cluster-nightly-4-21-4bm8nq: failed starting cluster creation "cluster-nightly-4-21-4bm8nq" in resourcegroup="rg-nightly-4-21-9xw8wplswg6p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-9xw8wplswg6p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-21-4bm8nq
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-21-9xw8wplswg6p/cluster-nightly-4-21-4bm8nq should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a941c0>: 
    failed to create HCP cluster cluster-nightly-4-21-4bm8nq: failed starting cluster creation "cluster-nightly-4-21-4bm8nq" in resourcegroup="rg-nightly-4-21-9xw8wplswg6p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-9xw8wplswg6p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-21-4bm8nq
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-c65rwd" at install version 4.22.0-0.nightly-multi-2026-09-23-151735
Unexpected error:
    <*fmt.wrapError | 0xc000e162c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-c65rwd: failed starting cluster creation "cp-ystream-upgrade-4-23-c65rwd" in resourcegroup="rg-cp-ystream-upgrade-4-23-xc5ljtzh9ws2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-xc5ljtzh9ws2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-c65rwd
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-c65rwd" at install version 4.22.0-0.nightly-multi-2026-09-23-151735
Unexpected error:
    <*fmt.wrapError | 0xc000e162c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-c65rwd: failed starting cluster creation "cp-ystream-upgrade-4-23-c65rwd" in resourcegroup="rg-cp-ystream-upgrade-4-23-xc5ljtzh9ws2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-xc5ljtzh9ws2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-c65rwd
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-vdm9fjw2dttb/cluster-nightly-5-1-v74km7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c25980>: 
    failed to create HCP cluster cluster-nightly-5-1-v74km7: failed starting cluster creation "cluster-nightly-5-1-v74km7" in resourcegroup="rg-nightly-5-1-vdm9fjw2dttb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-vdm9fjw2dttb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-1-v74km7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-5-1-vdm9fjw2dttb/cluster-nightly-5-1-v74km7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c25980>: 
    failed to create HCP cluster cluster-nightly-5-1-v74km7: failed starting cluster creation "cluster-nightly-5-1-v74km7" in resourcegroup="rg-nightly-5-1-vdm9fjw2dttb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-vdm9fjw2dttb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-1-v74km7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b182c0>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-p2sz9v5k8jtd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-p2sz9v5k8jtd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b182c0>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-p2sz9v5k8jtd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-p2sz9v5k8jtd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-bgpnp8wsw6xn/cluster-nightly-4-22-ld22vq should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f241c0>: 
    failed to create HCP cluster cluster-nightly-4-22-ld22vq: failed starting cluster creation "cluster-nightly-4-22-ld22vq" in resourcegroup="rg-nightly-4-22-bgpnp8wsw6xn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-bgpnp8wsw6xn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-22-ld22vq
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-22-bgpnp8wsw6xn/cluster-nightly-4-22-ld22vq should provision
Unexpected error:
    <*fmt.wrapError | 0xc000f241c0>: 
    failed to create HCP cluster cluster-nightly-4-22-ld22vq: failed starting cluster creation "cluster-nightly-4-22-ld22vq" in resourcegroup="rg-nightly-4-22-bgpnp8wsw6xn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-bgpnp8wsw6xn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-22-ld22vq
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-4zq985" at install version 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc00150e0e0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-4zq985: failed starting cluster creation "cp-ystream-upgrade-4-21-4zq985" in resourcegroup="rg-cp-ystream-upgrade-4-21-2t5stzd262x4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-2t5stzd262x4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-4zq985
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-4zq985" at install version 4.20.0-0.nightly-multi-2026-09-23-090319
Unexpected error:
    <*fmt.wrapError | 0xc00150e0e0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-4zq985: failed starting cluster creation "cp-ystream-upgrade-4-21-4zq985" in resourcegroup="rg-cp-ystream-upgrade-4-21-2t5stzd262x4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-2t5stzd262x4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-4zq985
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0010c8360>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-g7klgrs9fgjl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-g7klgrs9fgjl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0010c8360>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-g7klgrs9fgjl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-g7klgrs9fgjl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc001360160>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-zht9gtvbf76b": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zht9gtvbf76b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc001360160>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-zht9gtvbf76b": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-zht9gtvbf76b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000980380>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-xw7mgxvr7szh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-xw7mgxvr7szh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000980380>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-xw7mgxvr7szh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-xw7mgxvr7szh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0023de0e0>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-295h87tpx2wb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-295h87tpx2wb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0023de0e0>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-295h87tpx2wb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-295h87tpx2wb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e5d140>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-ljcwvrrqzgb6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-ljcwvrrqzgb6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e5d140>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-ljcwvrrqzgb6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-ljcwvrrqzgb6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000f64220>: 
    failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-nj4mbs7m2xnq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-nj4mbs7m2xnq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000f64220>: 
    failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-nj4mbs7m2xnq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-nj4mbs7m2xnq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc00101a240>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-tzwdzblv4lfz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-tzwdzblv4lfz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc00101a240>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-tzwdzblv4lfz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-tzwdzblv4lfz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000ab6240>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-s72gl49bdlhm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-s72gl49bdlhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000ab6240>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-s72gl49bdlhm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-s72gl49bdlhm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0010ac320>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-d2dfnkr542l8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-d2dfnkr542l8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0010ac320>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-d2dfnkr542l8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-d2dfnkr542l8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc000c46330>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-b62sjvpqvlrq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc000c46330>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-b62sjvpqvlrq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00016cbc0>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-57hgcc4bw6nt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-57hgcc4bw6nt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00016cbc0>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-57hgcc4bw6nt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-57hgcc4bw6nt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-skk87w with version 4.21.0-0.nightly-multi-2026-09-23-170748
Unexpected error:
    <*fmt.wrapError | 0xc000e62b20>: 
    failed to create HCP cluster np-version-upgrade-cluster-skk87w: failed starting cluster creation "np-version-upgrade-cluster-skk87w" in resourcegroup="rg-np-version-upgrade-skk87w-cq5mh9x4f6m8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-skk87w-cq5mh9x4f6m8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-skk87w
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-skk87w with version 4.21.0-0.nightly-multi-2026-09-23-170748
Unexpected error:
    <*fmt.wrapError | 0xc000e62b20>: 
    failed to create HCP cluster np-version-upgrade-cluster-skk87w: failed starting cluster creation "np-version-upgrade-cluster-skk87w" in resourcegroup="rg-np-version-upgrade-skk87w-cq5mh9x4f6m8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-skk87w-cq5mh9x4f6m8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-skk87w
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000bfc0e0>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-6klmwqw9sdwc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-6klmwqw9sdwc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000bfc0e0>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-6klmwqw9sdwc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-6klmwqw9sdwc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:101]: failed to begin creating HCP cluster "admin-cred-lifecycle-jjzc8b"
Unexpected error:
    <*fmt.wrapError | 0xc000761f00>: 
    failed starting cluster creation "admin-cred-lifecycle-jjzc8b" in resourcegroup="admin-credential-lifecycle-test-hb7c2sfbh46t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hb7c2sfbh46t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-jjzc8b
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:101]: failed to begin creating HCP cluster "admin-cred-lifecycle-jjzc8b"
Unexpected error:
    <*fmt.wrapError | 0xc000761f00>: 
    failed starting cluster creation "admin-cred-lifecycle-jjzc8b" in resourcegroup="admin-credential-lifecycle-test-hb7c2sfbh46t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-hb7c2sfbh46t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-jjzc8b
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000df0260>: 
    failed to create HCP cluster np-dg-minor-jcknfn: failed starting cluster creation "np-dg-minor-jcknfn" in resourcegroup="rg-np-dg-minor-jcknfn-jp7kllql2h9n": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-jcknfn-jp7kllql2h9n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-jcknfn
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000df0260>: 
    failed to create HCP cluster np-dg-minor-jcknfn: failed starting cluster creation "np-dg-minor-jcknfn" in resourcegroup="rg-np-dg-minor-jcknfn-jp7kllql2h9n": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-jcknfn-jp7kllql2h9n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-jcknfn
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-20-xflblvj4b79l/cluster-nightly-4-20-hp5qw6 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c8a340>: 
    failed to create HCP cluster cluster-nightly-4-20-hp5qw6: failed starting cluster creation "cluster-nightly-4-20-hp5qw6" in resourcegroup="rg-nightly-4-20-xflblvj4b79l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-xflblvj4b79l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-hp5qw6
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-nightly-4-20-xflblvj4b79l/cluster-nightly-4-20-hp5qw6 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c8a340>: 
    failed to create HCP cluster cluster-nightly-4-20-hp5qw6: failed starting cluster creation "cluster-nightly-4-20-hp5qw6" in resourcegroup="rg-nightly-4-20-xflblvj4b79l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-xflblvj4b79l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-hp5qw6
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000ea63a0>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-md9bxlt4h2bl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-md9bxlt4h2bl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000ea63a0>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-md9bxlt4h2bl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-md9bxlt4h2bl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000ac62a0>: 
    failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-kr4qllwsq4zw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-kr4qllwsq4zw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000ac62a0>: 
    failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-kr4qllwsq4zw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-kr4qllwsq4zw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0008a26c0>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-gwq5rnfvqbmj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-gwq5rnfvqbmj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0008a26c0>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-gwq5rnfvqbmj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-gwq5rnfvqbmj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:105]: failed to begin creating HCP cluster "admin-cred-lifecycle-f9mwlb"
Unexpected error:
    <*fmt.wrapError | 0xc00014aa00>: 
    failed starting cluster creation "admin-cred-lifecycle-f9mwlb" in resourcegroup="admin-credential-lifecycle-test-jnk69g62fsqq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-jnk69g62fsqq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-f9mwlb
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:105]: failed to begin creating HCP cluster "admin-cred-lifecycle-f9mwlb"
Unexpected error:
    <*fmt.wrapError | 0xc00014aa00>: 
    failed starting cluster creation "admin-cred-lifecycle-f9mwlb" in resourcegroup="admin-credential-lifecycle-test-jnk69g62fsqq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-jnk69g62fsqq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-f9mwlb
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-rdjnfw with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc0009944c0>: 
    failed to create HCP cluster np-version-upgrade-cluster-rdjnfw: failed starting cluster creation "np-version-upgrade-cluster-rdjnfw" in resourcegroup="rg-np-version-upgrade-rdjnfw-5m2htlhcktvw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-rdjnfw-5m2htlhcktvw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-rdjnfw
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-rdjnfw with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc0009944c0>: 
    failed to create HCP cluster np-version-upgrade-cluster-rdjnfw: failed starting cluster creation "np-version-upgrade-cluster-rdjnfw" in resourcegroup="rg-np-version-upgrade-rdjnfw-5m2htlhcktvw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-rdjnfw-5m2htlhcktvw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-rdjnfw
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-mr648lrwtr8j/cluster-candidate-5-0-jdjc8h should provision
Unexpected error:
    <*fmt.wrapError | 0xc001926140>: 
    failed to create HCP cluster cluster-candidate-5-0-jdjc8h: failed starting cluster creation "cluster-candidate-5-0-jdjc8h" in resourcegroup="rg-candidate-5-0-mr648lrwtr8j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mr648lrwtr8j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-jdjc8h
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-mr648lrwtr8j/cluster-candidate-5-0-jdjc8h should provision
Unexpected error:
    <*fmt.wrapError | 0xc001926140>: 
    failed to create HCP cluster cluster-candidate-5-0-jdjc8h: failed starting cluster creation "cluster-candidate-5-0-jdjc8h" in resourcegroup="rg-candidate-5-0-mr648lrwtr8j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mr648lrwtr8j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-jdjc8h
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000a7a320>: 
    failed to create HCP cluster ea-listdnmz6c: failed starting cluster creation "ea-listdnmz6c" in resourcegroup="ea-list-2tprnprxzp8c": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-2tprnprxzp8c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listdnmz6c
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000a7a320>: 
    failed to create HCP cluster ea-listdnmz6c: failed starting cluster creation "ea-listdnmz6c" in resourcegroup="ea-list-2tprnprxzp8c": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-2tprnprxzp8c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listdnmz6c
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000c34180>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-c2vqb28n2z6s": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-c2vqb28n2z6s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000c34180>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-c2vqb28n2z6s": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-c2vqb28n2z6s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc000a982d0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-gwwmf6gjgf55/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc000a982d0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-gwwmf6gjgf55/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-lmjw65" with version 4.22.14 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000fa6a20>: 
    failed to create HCP cluster cluster-zstream-4-22-lmjw65: failed starting cluster creation "cluster-zstream-4-22-lmjw65" in resourcegroup="rg-zstream-upgrade-4-22-79tsm6r6kxcv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-79tsm6r6kxcv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-lmjw65
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-lmjw65" with version 4.22.14 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000fa6a20>: 
    failed to create HCP cluster cluster-zstream-4-22-lmjw65: failed starting cluster creation "cluster-zstream-4-22-lmjw65" in resourcegroup="rg-zstream-upgrade-4-22-79tsm6r6kxcv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-79tsm6r6kxcv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-lmjw65
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc0010ea200>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-wc9p6hpc29b6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-wc9p6hpc29b6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc0010ea200>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-wc9p6hpc29b6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-wc9p6hpc29b6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00062c3e0>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-9j5894xv2nfk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-9j5894xv2nfk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00062c3e0>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-9j5894xv2nfk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-9j5894xv2nfk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000992780>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-qq5gvs64ckln": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-qq5gvs64ckln/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000992780>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-qq5gvs64ckln": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-qq5gvs64ckln/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc00081aa20>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-w2klxbbvrh6r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-w2klxbbvrh6r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc00081aa20>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-w2klxbbvrh6r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-w2klxbbvrh6r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ba4240>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-49bvw77ddhd7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-49bvw77ddhd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ba4240>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-49bvw77ddhd7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-49bvw77ddhd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc001310120>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-nwsnnfvwzdrd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-nwsnnfvwzdrd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc001310120>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-nwsnnfvwzdrd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-nwsnnfvwzdrd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-t6w7kn with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc001586240>: 
    failed to create HCP cluster np-version-upgrade-cluster-t6w7kn: failed starting cluster creation "np-version-upgrade-cluster-t6w7kn" in resourcegroup="rg-np-version-upgrade-t6w7kn-n527hph4clhc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-t6w7kn-n527hph4clhc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-t6w7kn
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-t6w7kn with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc001586240>: 
    failed to create HCP cluster np-version-upgrade-cluster-t6w7kn: failed starting cluster creation "np-version-upgrade-cluster-t6w7kn" in resourcegroup="rg-np-version-upgrade-t6w7kn-n527hph4clhc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-t6w7kn-n527hph4clhc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-t6w7kn
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-zprvbm" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc0004d2fa0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-zprvbm: failed starting cluster creation "cp-ystream-upgrade-4-23-zprvbm" in resourcegroup="rg-cp-ystream-upgrade-4-23-nc75hl29zdtj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-nc75hl29zdtj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-zprvbm
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-zprvbm" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc0004d2fa0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-zprvbm: failed starting cluster creation "cp-ystream-upgrade-4-23-zprvbm" in resourcegroup="rg-cp-ystream-upgrade-4-23-nc75hl29zdtj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-nc75hl29zdtj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-zprvbm
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000d385c0>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-b2v8h4lk7mr9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-b2v8h4lk7mr9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000d385c0>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-b2v8h4lk7mr9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-b2v8h4lk7mr9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00143a1e0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-m7td24fcmrl7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-m7td24fcmrl7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00143a1e0>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-m7td24fcmrl7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-m7td24fcmrl7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000506300>: 
    failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-8dfwnhhtxdp6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8dfwnhhtxdp6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000506300>: 
    failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-8dfwnhhtxdp6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-8dfwnhhtxdp6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-6tlfjxd6xqc6/cluster-candidate-4-22-xvfpvx should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fba3a0>: 
    failed to create HCP cluster cluster-candidate-4-22-xvfpvx: failed starting cluster creation "cluster-candidate-4-22-xvfpvx" in resourcegroup="rg-candidate-4-22-6tlfjxd6xqc6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-6tlfjxd6xqc6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-22-xvfpvx
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-6tlfjxd6xqc6/cluster-candidate-4-22-xvfpvx should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fba3a0>: 
    failed to create HCP cluster cluster-candidate-4-22-xvfpvx: failed starting cluster creation "cluster-candidate-4-22-xvfpvx" in resourcegroup="rg-candidate-4-22-6tlfjxd6xqc6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-6tlfjxd6xqc6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-22-xvfpvx
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00123c100>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-zhdp8544srss": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-zhdp8544srss/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00123c100>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-zhdp8544srss": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-zhdp8544srss/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-66tjf2" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc0011c0240>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-66tjf2: failed starting cluster creation "cp-ystream-upgrade-4-21-66tjf2" in resourcegroup="rg-cp-ystream-upgrade-4-21-fp5kzmg9k2qb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-fp5kzmg9k2qb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-66tjf2
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-66tjf2" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc0011c0240>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-66tjf2: failed starting cluster creation "cp-ystream-upgrade-4-21-66tjf2" in resourcegroup="rg-cp-ystream-upgrade-4-21-fp5kzmg9k2qb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-fp5kzmg9k2qb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-66tjf2
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-f8mhn9" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc000f900e0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-f8mhn9: failed starting cluster creation "cp-ystream-upgrade-5-0-f8mhn9" in resourcegroup="rg-cp-ystream-upgrade-5-0-mf5jqrbn6srb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-mf5jqrbn6srb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-f8mhn9
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-f8mhn9" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc000f900e0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-f8mhn9: failed starting cluster creation "cp-ystream-upgrade-5-0-f8mhn9" in resourcegroup="rg-cp-ystream-upgrade-5-0-mf5jqrbn6srb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-mf5jqrbn6srb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-f8mhn9
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-kf4fc8" with version 4.20.38 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000e3e620>: 
    failed to create HCP cluster cluster-zstream-4-20-kf4fc8: failed starting cluster creation "cluster-zstream-4-20-kf4fc8" in resourcegroup="rg-zstream-upgrade-4-20-24x8x7qddc4f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-24x8x7qddc4f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-kf4fc8
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-kf4fc8" with version 4.20.38 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000e3e620>: 
    failed to create HCP cluster cluster-zstream-4-20-kf4fc8: failed starting cluster creation "cluster-zstream-4-20-kf4fc8" in resourcegroup="rg-zstream-upgrade-4-20-24x8x7qddc4f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-24x8x7qddc4f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-kf4fc8
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-23-hqj25xm5xjpp/cluster-candidate-4-23-x87dzg should provision
Unexpected error:
    <*fmt.wrapError | 0xc001010300>: 
    failed to create HCP cluster cluster-candidate-4-23-x87dzg: failed starting cluster creation "cluster-candidate-4-23-x87dzg" in resourcegroup="rg-candidate-4-23-hqj25xm5xjpp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-hqj25xm5xjpp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-23-x87dzg
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-23-hqj25xm5xjpp/cluster-candidate-4-23-x87dzg should provision
Unexpected error:
    <*fmt.wrapError | 0xc001010300>: 
    failed to create HCP cluster cluster-candidate-4-23-x87dzg: failed starting cluster creation "cluster-candidate-4-23-x87dzg" in resourcegroup="rg-candidate-4-23-hqj25xm5xjpp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-23-hqj25xm5xjpp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-23-x87dzg
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc0008123c0>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-49kfk2h5cd4x": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-49kfk2h5cd4x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc0008123c0>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-49kfk2h5cd4x": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-49kfk2h5cd4x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000f42380>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-744mhb4wmcsp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-744mhb4wmcsp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000f42380>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-744mhb4wmcsp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-744mhb4wmcsp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ea2120>: 
    failed to create HCP cluster np-dg-minor-5xtqzm: failed starting cluster creation "np-dg-minor-5xtqzm" in resourcegroup="rg-np-dg-minor-5xtqzm-28nz5th2j6m7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-5xtqzm-28nz5th2j6m7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-5xtqzm
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ea2120>: 
    failed to create HCP cluster np-dg-minor-5xtqzm: failed starting cluster creation "np-dg-minor-5xtqzm" in resourcegroup="rg-np-dg-minor-5xtqzm-28nz5th2j6m7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-5xtqzm-28nz5th2j6m7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-5xtqzm
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b9c260>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-wftkcwprcffb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-wftkcwprcffb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b9c260>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-wftkcwprcffb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-wftkcwprcffb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000aa6340>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-89xxsz542trh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-89xxsz542trh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000aa6340>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-89xxsz542trh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-89xxsz542trh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000788a60>: 
    failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-skgrd5ch2wdr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-skgrd5ch2wdr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000788a60>: 
    failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-skgrd5ch2wdr": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-skgrd5ch2wdr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-jcpfwd at version 4.22.15
Unexpected error:
    <*fmt.wrapError | 0xc001808280>: 
    failed to create HCP cluster np-skip-minor-jcpfwd: failed starting cluster creation "np-skip-minor-jcpfwd" in resourcegroup="rg-np-skip-minor-jcpfwd-gvqsdxzjh8vp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-jcpfwd-gvqsdxzjh8vp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-jcpfwd
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-jcpfwd at version 4.22.15
Unexpected error:
    <*fmt.wrapError | 0xc001808280>: 
    failed to create HCP cluster np-skip-minor-jcpfwd: failed starting cluster creation "np-skip-minor-jcpfwd" in resourcegroup="rg-np-skip-minor-jcpfwd-gvqsdxzjh8vp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-jcpfwd-gvqsdxzjh8vp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-jcpfwd
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc00154a810>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc00154a810>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc0014940e0>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-2pqz77p5k645": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-2pqz77p5k645/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc0014940e0>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-2pqz77p5k645": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-2pqz77p5k645/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000614500>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-hwvkqfn2c69n": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-hwvkqfn2c69n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000614500>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-hwvkqfn2c69n": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-hwvkqfn2c69n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc001002280>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-sp7x4pd4zkrt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-sp7x4pd4zkrt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc001002280>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-sp7x4pd4zkrt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-sp7x4pd4zkrt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-7pqfcm with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc00134e120>: 
    failed to create HCP cluster np-version-upgrade-cluster-7pqfcm: failed starting cluster creation "np-version-upgrade-cluster-7pqfcm" in resourcegroup="rg-np-version-upgrade-7pqfcm-2zqnz4g9wfx5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7pqfcm-2zqnz4g9wfx5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-7pqfcm
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-7pqfcm with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc00134e120>: 
    failed to create HCP cluster np-version-upgrade-cluster-7pqfcm: failed starting cluster creation "np-version-upgrade-cluster-7pqfcm" in resourcegroup="rg-np-version-upgrade-7pqfcm-2zqnz4g9wfx5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7pqfcm-2zqnz4g9wfx5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-7pqfcm
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-nr97qpqbnq89/cluster-candidate-4-21-vbtv2b should provision
Unexpected error:
    <*fmt.wrapError | 0xc000914640>: 
    failed to create HCP cluster cluster-candidate-4-21-vbtv2b: failed starting cluster creation "cluster-candidate-4-21-vbtv2b" in resourcegroup="rg-candidate-4-21-nr97qpqbnq89": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-nr97qpqbnq89/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-21-vbtv2b
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-nr97qpqbnq89/cluster-candidate-4-21-vbtv2b should provision
Unexpected error:
    <*fmt.wrapError | 0xc000914640>: 
    failed to create HCP cluster cluster-candidate-4-21-vbtv2b: failed starting cluster creation "cluster-candidate-4-21-vbtv2b" in resourcegroup="rg-candidate-4-21-nr97qpqbnq89": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-nr97qpqbnq89/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-21-vbtv2b
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000a49080>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-grmp56x8mfgq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-grmp56x8mfgq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000a49080>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-grmp56x8mfgq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-grmp56x8mfgq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000ca01e0>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-lzr7ggchsgl8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-lzr7ggchsgl8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000ca01e0>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-lzr7ggchsgl8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-lzr7ggchsgl8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00098c6c0>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-dzql26rplxsd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-dzql26rplxsd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00098c6c0>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-dzql26rplxsd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-dzql26rplxsd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc001048260>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-n85wjx2ql25d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-n85wjx2ql25d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc001048260>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-n85wjx2ql25d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-n85wjx2ql25d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-7pqq8c" with version 5.0.0-rc.2 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000a2cd60>: 
    failed to create HCP cluster cluster-zstream-5-0-7pqq8c: failed starting cluster creation "cluster-zstream-5-0-7pqq8c" in resourcegroup="rg-zstream-upgrade-5-0-psl5tsd9krt8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-psl5tsd9krt8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-7pqq8c
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-7pqq8c" with version 5.0.0-rc.2 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000a2cd60>: 
    failed to create HCP cluster cluster-zstream-5-0-7pqq8c: failed starting cluster creation "cluster-zstream-5-0-7pqq8c" in resourcegroup="rg-zstream-upgrade-5-0-psl5tsd9krt8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-psl5tsd9krt8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-7pqq8c
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc0006871a0>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-npl4lqlc2j28": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-npl4lqlc2j28/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc0006871a0>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-npl4lqlc2j28": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-npl4lqlc2j28/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000fe01e0>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-k5wx2hsgkgzp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-k5wx2hsgkgzp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000fe01e0>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-k5wx2hsgkgzp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-k5wx2hsgkgzp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-24kkdn" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000789c60>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-24kkdn: failed starting cluster creation "cp-ystream-upgrade-4-22-24kkdn" in resourcegroup="rg-cp-ystream-upgrade-4-22-jchspf48txds": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-jchspf48txds/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-24kkdn
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-24kkdn" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc000789c60>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-24kkdn: failed starting cluster creation "cp-ystream-upgrade-4-22-24kkdn" in resourcegroup="rg-cp-ystream-upgrade-4-22-jchspf48txds": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-jchspf48txds/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-24kkdn
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-rfcvwdhs2s8z/cluster-candidate-4-20-fb65qk should provision
Unexpected error:
    <*fmt.wrapError | 0xc000911220>: 
    failed to create HCP cluster cluster-candidate-4-20-fb65qk: failed starting cluster creation "cluster-candidate-4-20-fb65qk" in resourcegroup="rg-candidate-4-20-rfcvwdhs2s8z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rfcvwdhs2s8z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-20-fb65qk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-rfcvwdhs2s8z/cluster-candidate-4-20-fb65qk should provision
Unexpected error:
    <*fmt.wrapError | 0xc000911220>: 
    failed to create HCP cluster cluster-candidate-4-20-fb65qk: failed starting cluster creation "cluster-candidate-4-20-fb65qk" in resourcegroup="rg-candidate-4-20-rfcvwdhs2s8z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-rfcvwdhs2s8z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-20-fb65qk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-5m6fvn" with version 4.21.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00106c220>: 
    failed to create HCP cluster cluster-zstream-4-21-5m6fvn: failed starting cluster creation "cluster-zstream-4-21-5m6fvn" in resourcegroup="rg-zstream-upgrade-4-21-585ggz8lqzc9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-585ggz8lqzc9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-5m6fvn
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-5m6fvn" with version 4.21.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc00106c220>: 
    failed to create HCP cluster cluster-zstream-4-21-5m6fvn: failed starting cluster creation "cluster-zstream-4-21-5m6fvn" in resourcegroup="rg-zstream-upgrade-4-21-585ggz8lqzc9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-585ggz8lqzc9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-5m6fvn
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000bd01a0>: 
    failed to create HCP cluster np-downgrade-j4m7th: failed starting cluster creation "np-downgrade-j4m7th" in resourcegroup="rg-np-downgrade-j4m7th-k6x7c5vhdzqw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-j4m7th-k6x7c5vhdzqw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-j4m7th
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000bd01a0>: 
    failed to create HCP cluster np-downgrade-j4m7th: failed starting cluster creation "np-downgrade-j4m7th" in resourcegroup="rg-np-downgrade-j4m7th-k6x7c5vhdzqw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-j4m7th-k6x7c5vhdzqw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-j4m7th
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001690100>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-mxsb4d4dcndm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-mxsb4d4dcndm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001690100>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-mxsb4d4dcndm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-mxsb4d4dcndm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000946140>: 
    failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-gxzd4bqsq7ff": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-gxzd4bqsq7ff/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000946140>: 
    failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-gxzd4bqsq7ff": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-gxzd4bqsq7ff/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc001044400>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-8g9xp5jpnqwl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-8g9xp5jpnqwl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc001044400>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-8g9xp5jpnqwl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-8g9xp5jpnqwl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000cde9a0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-7gkqvtjn5hrh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7gkqvtjn5hrh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000cde9a0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-7gkqvtjn5hrh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-7gkqvtjn5hrh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-qd9tq7
Unexpected error:
    <*fmt.wrapError | 0xc000d82360>: 
    failed to create HCP cluster np-noedge-qd9tq7: failed starting cluster creation "np-noedge-qd9tq7" in resourcegroup="rg-np-noedge-qd9tq7-j6nd68sppgnf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-qd9tq7-j6nd68sppgnf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-qd9tq7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-qd9tq7
Unexpected error:
    <*fmt.wrapError | 0xc000d82360>: 
    failed to create HCP cluster np-noedge-qd9tq7: failed starting cluster creation "np-noedge-qd9tq7" in resourcegroup="rg-np-noedge-qd9tq7-j6nd68sppgnf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-qd9tq7-j6nd68sppgnf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-qd9tq7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000a92300>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-zn5n8g9qjq6p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-zn5n8g9qjq6p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc000a92300>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-zn5n8g9qjq6p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-zn5n8g9qjq6p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009582a0>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-gggk2qcjxcn8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gggk2qcjxcn8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0009582a0>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-gggk2qcjxcn8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-gggk2qcjxcn8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000d57840>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-j4n5xfxkstwg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-j4n5xfxkstwg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000d57840>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-j4n5xfxkstwg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-j4n5xfxkstwg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc0005dc160>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-22256vxjwxzv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-22256vxjwxzv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc0005dc160>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-22256vxjwxzv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-22256vxjwxzv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc0005a4440>: 
    failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-g2r7df4nbvtl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-g2r7df4nbvtl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc0005a4440>: 
    failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-g2r7df4nbvtl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-g2r7df4nbvtl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0005886e0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-tnkkshhqpgpc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-tnkkshhqpgpc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc0005886e0>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-tnkkshhqpgpc": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-tnkkshhqpgpc/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc0007ef1a0>: 
    failed to create HCP cluster ea-listmfk6rl: failed starting cluster creation "ea-listmfk6rl" in resourcegroup="ea-list-bkxpqlfv4c6z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-bkxpqlfv4c6z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listmfk6rl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc0007ef1a0>: 
    failed to create HCP cluster ea-listmfk6rl: failed starting cluster creation "ea-listmfk6rl" in resourcegroup="ea-list-bkxpqlfv4c6z": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-bkxpqlfv4c6z/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-listmfk6rl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc0001a2500>: 
    failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-fwgpzzwmqdsk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-fwgpzzwmqdsk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc0001a2500>: 
    failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-fwgpzzwmqdsk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-fwgpzzwmqdsk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc00067a300>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-rsb2d87wnkmj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-rsb2d87wnkmj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc00067a300>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-rsb2d87wnkmj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-rsb2d87wnkmj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-wcfsw2 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc00184a100>: 
    failed to create HCP cluster np-version-upgrade-cluster-wcfsw2: failed starting cluster creation "np-version-upgrade-cluster-wcfsw2" in resourcegroup="rg-np-version-upgrade-wcfsw2-tgvfvwtvq9vl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-wcfsw2-tgvfvwtvq9vl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-wcfsw2
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-wcfsw2 with version 4.20.39
Unexpected error:
    <*fmt.wrapError | 0xc00184a100>: 
    failed to create HCP cluster np-version-upgrade-cluster-wcfsw2: failed starting cluster creation "np-version-upgrade-cluster-wcfsw2" in resourcegroup="rg-np-version-upgrade-wcfsw2-tgvfvwtvq9vl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-wcfsw2-tgvfvwtvq9vl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-wcfsw2
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000ec4100>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-5zhpbqbl9m9q": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-5zhpbqbl9m9q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000ec4100>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-5zhpbqbl9m9q": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-5zhpbqbl9m9q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-h89dm9lhsl2f/cluster-candidate-4-22-w9djqt should provision
Unexpected error:
    <*fmt.wrapError | 0xc00110a1a0>: 
    failed to create HCP cluster cluster-candidate-4-22-w9djqt: failed starting cluster creation "cluster-candidate-4-22-w9djqt" in resourcegroup="rg-candidate-4-22-h89dm9lhsl2f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-h89dm9lhsl2f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-22-w9djqt
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-22-h89dm9lhsl2f/cluster-candidate-4-22-w9djqt should provision
Unexpected error:
    <*fmt.wrapError | 0xc00110a1a0>: 
    failed to create HCP cluster cluster-candidate-4-22-w9djqt: failed starting cluster creation "cluster-candidate-4-22-w9djqt" in resourcegroup="rg-candidate-4-22-h89dm9lhsl2f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-h89dm9lhsl2f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-22-w9djqt
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-95hssq" with version 4.22.14 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b24480>: 
    failed to create HCP cluster cluster-zstream-4-22-95hssq: failed starting cluster creation "cluster-zstream-4-22-95hssq" in resourcegroup="rg-zstream-upgrade-4-22-mw82tvb5wt8f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-mw82tvb5wt8f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-95hssq
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-95hssq" with version 4.22.14 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b24480>: 
    failed to create HCP cluster cluster-zstream-4-22-95hssq: failed starting cluster creation "cluster-zstream-4-22-95hssq" in resourcegroup="rg-zstream-upgrade-4-22-mw82tvb5wt8f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-mw82tvb5wt8f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-95hssq
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000e12740>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-fnhqkgb28q5j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-fnhqkgb28q5j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000e12740>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-fnhqkgb28q5j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-fnhqkgb28q5j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000a841e0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2pscx6lt6hbw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2pscx6lt6hbw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000a841e0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2pscx6lt6hbw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2pscx6lt6hbw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c94200>: 
    failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-qhgbvbhnhp7c": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-qhgbvbhnhp7c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c94200>: 
    failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-qhgbvbhnhp7c": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-qhgbvbhnhp7c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc000c00780>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7q6zb9qhrlww": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-7q6zb9qhrlww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc000c00780>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-7q6zb9qhrlww": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-7q6zb9qhrlww/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc001026160>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-s8j9mmmntb7s": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-s8j9mmmntb7s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc001026160>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-s8j9mmmntb7s": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-s8j9mmmntb7s/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00084b4c0>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-vjfmvmg6928d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-vjfmvmg6928d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00084b4c0>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-vjfmvmg6928d": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-vjfmvmg6928d/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001012280>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4lj84nw2x245": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4lj84nw2x245/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:81]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001012280>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-4lj84nw2x245": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-4lj84nw2x245/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc001380240>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-8xv96lzbm5cx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-8xv96lzbm5cx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc001380240>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-8xv96lzbm5cx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-8xv96lzbm5cx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc00090d780>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-4znnjj2bvmhl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-4znnjj2bvmhl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc00090d780>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-4znnjj2bvmhl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-4znnjj2bvmhl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc00079e0e0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-lpzbtxsqnsjm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-lpzbtxsqnsjm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc00079e0e0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-lpzbtxsqnsjm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-lpzbtxsqnsjm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0012a2120>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-fpsrssl7cjkj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-fpsrssl7cjkj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0012a2120>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-fpsrssl7cjkj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-fpsrssl7cjkj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f02640>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-svfw267gk9r2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-svfw267gk9r2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f02640>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-svfw267gk9r2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-svfw267gk9r2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000b66340>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-q4smrwpc7lth": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-q4smrwpc7lth/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000b66340>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-q4smrwpc7lth": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-q4smrwpc7lth/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-6zw86fclhnhh/cluster-candidate-4-20-7rdtb7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ff62a0>: 
    failed to create HCP cluster cluster-candidate-4-20-7rdtb7: failed starting cluster creation "cluster-candidate-4-20-7rdtb7" in resourcegroup="rg-candidate-4-20-6zw86fclhnhh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-6zw86fclhnhh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-20-7rdtb7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-20-6zw86fclhnhh/cluster-candidate-4-20-7rdtb7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc000ff62a0>: 
    failed to create HCP cluster cluster-candidate-4-20-7rdtb7: failed starting cluster creation "cluster-candidate-4-20-7rdtb7" in resourcegroup="rg-candidate-4-20-6zw86fclhnhh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-6zw86fclhnhh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-20-7rdtb7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-gk8mbp" with version 4.21.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000d5e380>: 
    failed to create HCP cluster cluster-zstream-4-21-gk8mbp: failed starting cluster creation "cluster-zstream-4-21-gk8mbp" in resourcegroup="rg-zstream-upgrade-4-21-k25gftsdpj5n": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-k25gftsdpj5n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk8mbp
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-gk8mbp" with version 4.21.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000d5e380>: 
    failed to create HCP cluster cluster-zstream-4-21-gk8mbp: failed starting cluster creation "cluster-zstream-4-21-gk8mbp" in resourcegroup="rg-zstream-upgrade-4-21-k25gftsdpj5n": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-k25gftsdpj5n/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-gk8mbp
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d32140>: 
    failed to create HCP cluster np-dg-minor-89j949: failed starting cluster creation "np-dg-minor-89j949" in resourcegroup="rg-np-dg-minor-89j949-ftfm8c66f846": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-89j949-ftfm8c66f846/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-89j949
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d32140>: 
    failed to create HCP cluster np-dg-minor-89j949: failed starting cluster creation "np-dg-minor-89j949" in resourcegroup="rg-np-dg-minor-89j949-ftfm8c66f846": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-89j949-ftfm8c66f846/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-89j949
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0028ba100>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-np755vjcrftf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-np755vjcrftf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0028ba100>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-np755vjcrftf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-np755vjcrftf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0028b02a0>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-rrgpx55nq5gt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rrgpx55nq5gt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:208]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc0028b02a0>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-rrgpx55nq5gt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-rrgpx55nq5gt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000e5e360>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-79r5dzshjfjd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-79r5dzshjfjd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000e5e360>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-79r5dzshjfjd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-79r5dzshjfjd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001526320>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-xwmd8q84hgkl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-xwmd8q84hgkl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:124]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001526320>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-xwmd8q84hgkl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-xwmd8q84hgkl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-bnwsnt
Unexpected error:
    <*fmt.wrapError | 0xc0015e3040>: 
    failed to create HCP cluster np-noedge-bnwsnt: failed starting cluster creation "np-noedge-bnwsnt" in resourcegroup="rg-np-noedge-bnwsnt-dgc7qmfcddg5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-bnwsnt-dgc7qmfcddg5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-bnwsnt
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-bnwsnt
Unexpected error:
    <*fmt.wrapError | 0xc0015e3040>: 
    failed to create HCP cluster np-noedge-bnwsnt: failed starting cluster creation "np-noedge-bnwsnt" in resourcegroup="rg-np-noedge-bnwsnt-dgc7qmfcddg5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-bnwsnt-dgc7qmfcddg5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-bnwsnt
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-9s6lkg" with version 4.20.38 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000ef5020>: 
    failed to create HCP cluster cluster-zstream-4-20-9s6lkg: failed starting cluster creation "cluster-zstream-4-20-9s6lkg" in resourcegroup="rg-zstream-upgrade-4-20-42cxz7mjjbj2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-42cxz7mjjbj2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-9s6lkg
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-9s6lkg" with version 4.20.38 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000ef5020>: 
    failed to create HCP cluster cluster-zstream-4-20-9s6lkg: failed starting cluster creation "cluster-zstream-4-20-9s6lkg" in resourcegroup="rg-zstream-upgrade-4-20-42cxz7mjjbj2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-42cxz7mjjbj2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-9s6lkg
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-tjdjx7" with version 5.0.0-rc.2 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0008c4440>: 
    failed to create HCP cluster cluster-zstream-5-0-tjdjx7: failed starting cluster creation "cluster-zstream-5-0-tjdjx7" in resourcegroup="rg-zstream-upgrade-5-0-vzh67fzgfvck": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-vzh67fzgfvck/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-tjdjx7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-tjdjx7" with version 5.0.0-rc.2 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0008c4440>: 
    failed to create HCP cluster cluster-zstream-5-0-tjdjx7: failed starting cluster creation "cluster-zstream-5-0-tjdjx7" in resourcegroup="rg-zstream-upgrade-5-0-vzh67fzgfvck": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-vzh67fzgfvck/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-tjdjx7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000df82e0>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-6b525qt6f6jj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-6b525qt6f6jj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:99]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000df82e0>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-6b525qt6f6jj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-6b525qt6f6jj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0014e6140>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-mgsrktnswwmv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-mgsrktnswwmv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0014e6140>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-mgsrktnswwmv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-mgsrktnswwmv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-nqjgxv with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc0008d8220>: 
    failed to create HCP cluster np-version-upgrade-cluster-nqjgxv: failed starting cluster creation "np-version-upgrade-cluster-nqjgxv" in resourcegroup="rg-np-version-upgrade-nqjgxv-f2wzh798npkj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-nqjgxv-f2wzh798npkj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-nqjgxv
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-nqjgxv with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc0008d8220>: 
    failed to create HCP cluster np-version-upgrade-cluster-nqjgxv: failed starting cluster creation "np-version-upgrade-cluster-nqjgxv" in resourcegroup="rg-np-version-upgrade-nqjgxv-f2wzh798npkj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-nqjgxv-f2wzh798npkj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-nqjgxv
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001b19a80>: 
    failed to create HCP cluster np-downgrade-ppzl59: failed starting cluster creation "np-downgrade-ppzl59" in resourcegroup="rg-np-downgrade-ppzl59-6bp5w8kqrdc5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-ppzl59-6bp5w8kqrdc5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-ppzl59
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001b19a80>: 
    failed to create HCP cluster np-downgrade-ppzl59: failed starting cluster creation "np-downgrade-ppzl59" in resourcegroup="rg-np-downgrade-ppzl59-6bp5w8kqrdc5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-ppzl59-6bp5w8kqrdc5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-ppzl59
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-w95bj9 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000d0c0e0>: 
    failed to create HCP cluster np-version-upgrade-cluster-w95bj9: failed starting cluster creation "np-version-upgrade-cluster-w95bj9" in resourcegroup="rg-np-version-upgrade-w95bj9-bzlsbjrftkk4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w95bj9-bzlsbjrftkk4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w95bj9
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-w95bj9 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000d0c0e0>: 
    failed to create HCP cluster np-version-upgrade-cluster-w95bj9: failed starting cluster creation "np-version-upgrade-cluster-w95bj9" in resourcegroup="rg-np-version-upgrade-w95bj9-bzlsbjrftkk4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-w95bj9-bzlsbjrftkk4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-w95bj9
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-9rlflwwnznkv/cluster-candidate-4-21-xhlsgk should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a52440>: 
    failed to create HCP cluster cluster-candidate-4-21-xhlsgk: failed starting cluster creation "cluster-candidate-4-21-xhlsgk" in resourcegroup="rg-candidate-4-21-9rlflwwnznkv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-9rlflwwnznkv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-21-xhlsgk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-4-21-9rlflwwnznkv/cluster-candidate-4-21-xhlsgk should provision
Unexpected error:
    <*fmt.wrapError | 0xc000a52440>: 
    failed to create HCP cluster cluster-candidate-4-21-xhlsgk: failed starting cluster creation "cluster-candidate-4-21-xhlsgk" in resourcegroup="rg-candidate-4-21-9rlflwwnznkv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-9rlflwwnznkv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-21-xhlsgk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-vsjwr4 at version 4.22.15
Unexpected error:
    <*fmt.wrapError | 0xc000a2c700>: 
    failed to create HCP cluster np-skip-minor-vsjwr4: failed starting cluster creation "np-skip-minor-vsjwr4" in resourcegroup="rg-np-skip-minor-vsjwr4-bsn56prrsfv6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-vsjwr4-bsn56prrsfv6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-vsjwr4
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-vsjwr4 at version 4.22.15
Unexpected error:
    <*fmt.wrapError | 0xc000a2c700>: 
    failed to create HCP cluster np-skip-minor-vsjwr4: failed starting cluster creation "np-skip-minor-vsjwr4" in resourcegroup="rg-np-skip-minor-vsjwr4-bsn56prrsfv6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-vsjwr4-bsn56prrsfv6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-vsjwr4
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:91]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0010f2180>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-hxl9zp97mlgn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-hxl9zp97mlgn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:91]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0010f2180>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-hxl9zp97mlgn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-hxl9zp97mlgn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc001040140>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-t4djcjjxltjt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-t4djcjjxltjt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc001040140>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-t4djcjjxltjt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-t4djcjjxltjt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-tgns2n" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc0015630c0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-tgns2n: failed starting cluster creation "cp-ystream-upgrade-5-0-tgns2n" in resourcegroup="rg-cp-ystream-upgrade-5-0-rxzl7kxnnt9c": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-rxzl7kxnnt9c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-tgns2n
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-tgns2n" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc0015630c0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-tgns2n: failed starting cluster creation "cp-ystream-upgrade-5-0-tgns2n" in resourcegroup="rg-cp-ystream-upgrade-5-0-rxzl7kxnnt9c": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-rxzl7kxnnt9c/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-tgns2n
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000a38260>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-cqdlgmxn4w4v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-cqdlgmxn4w4v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc000a38260>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-cqdlgmxn4w4v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-cqdlgmxn4w4v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000564120>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-z7q4w677jrcv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-z7q4w677jrcv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000564120>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-z7q4w677jrcv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-z7q4w677jrcv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001002280>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-9zmvzjspzdbt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-9zmvzjspzdbt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001002280>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-9zmvzjspzdbt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-9zmvzjspzdbt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-tpgm4r" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc0018b0120>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-tpgm4r: failed starting cluster creation "cp-ystream-upgrade-4-22-tpgm4r" in resourcegroup="rg-cp-ystream-upgrade-4-22-mgqcx8h7xftx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-mgqcx8h7xftx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-tpgm4r
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-tpgm4r" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc0018b0120>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-tpgm4r: failed starting cluster creation "cp-ystream-upgrade-4-22-tpgm4r" in resourcegroup="rg-cp-ystream-upgrade-4-22-mgqcx8h7xftx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-mgqcx8h7xftx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-tpgm4r
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc0018261e0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-qh5rmkcjlrnr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc0018261e0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-qh5rmkcjlrnr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc00156e5a0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc00156e5a0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000700140>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-nqkrft7rwghd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-nqkrft7rwghd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:107]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000700140>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-nqkrft7rwghd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-nqkrft7rwghd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:103]: failed to begin creating HCP cluster "admin-cred-lifecycle-962qjz"
Unexpected error:
    <*fmt.wrapError | 0xc0010fa1e0>: 
    failed starting cluster creation "admin-cred-lifecycle-962qjz" in resourcegroup="admin-credential-lifecycle-test-7znlrnsb4d2v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-7znlrnsb4d2v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-962qjz
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:103]: failed to begin creating HCP cluster "admin-cred-lifecycle-962qjz"
Unexpected error:
    <*fmt.wrapError | 0xc0010fa1e0>: 
    failed starting cluster creation "admin-cred-lifecycle-962qjz" in resourcegroup="admin-credential-lifecycle-test-7znlrnsb4d2v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-7znlrnsb4d2v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-962qjz
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc001010120>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-qgdl4c87lvcl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-qgdl4c87lvcl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc001010120>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-qgdl4c87lvcl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-qgdl4c87lvcl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d823c0>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-gf5rj7zc4bks": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-gf5rj7zc4bks/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000d823c0>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-gf5rj7zc4bks": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-gf5rj7zc4bks/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-bv2pbs" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000a3e1c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-bv2pbs: failed starting cluster creation "cp-ystream-upgrade-4-21-bv2pbs" in resourcegroup="rg-cp-ystream-upgrade-4-21-x9l8w25mj4c7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-x9l8w25mj4c7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-bv2pbs
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-bv2pbs" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc000a3e1c0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-bv2pbs: failed starting cluster creation "cp-ystream-upgrade-4-21-bv2pbs" in resourcegroup="rg-cp-ystream-upgrade-4-21-x9l8w25mj4c7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-x9l8w25mj4c7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-bv2pbs
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-zf2lwttkqm8b/cluster-candidate-5-0-sfstjx should provision
Unexpected error:
    <*fmt.wrapError | 0xc000aa2160>: 
    failed to create HCP cluster cluster-candidate-5-0-sfstjx: failed starting cluster creation "cluster-candidate-5-0-sfstjx" in resourcegroup="rg-candidate-5-0-zf2lwttkqm8b": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-zf2lwttkqm8b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-sfstjx
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:123]: HCP cluster rg-candidate-5-0-zf2lwttkqm8b/cluster-candidate-5-0-sfstjx should provision
Unexpected error:
    <*fmt.wrapError | 0xc000aa2160>: 
    failed to create HCP cluster cluster-candidate-5-0-sfstjx: failed starting cluster creation "cluster-candidate-5-0-sfstjx" in resourcegroup="rg-candidate-5-0-zf2lwttkqm8b": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-zf2lwttkqm8b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-sfstjx
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred