<
Single Day: Sep 19▾
Pick a day
>
View JSON API
↗ Open Failure patterns for this day

Environment: DEV

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
1h 40m 31s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 42b2541 · merge 42b2541
canadacentral alert failed ★#7019 (merged) 1
alert (1)
alert [svc] FrontendPathLatency fired
22:43:02 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathLatency does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T23:51:12Z
  Ended:       2026-09-20T00:04:13Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j1465216-svc", component="frontend", method="get", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters exceeded 1 second over the past 30 minutes (cluster ci01-j1465216-svc).
2h 6m 3s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head d450510 · merge d450510
westus3 n/a passed #7021 (merged) 0 n/a
1h 43m 6s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 7a275cf · merge 7a275cf
canadacentral alert failed ★#7024 (merged) 1
alert (1)
alert [svc] FrontendPathMedianLatency fired
22:31:14 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T23:35:51Z
  Ended:       2026-09-19T23:41:50Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j6722432-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j6722432-svc).
1h 58m 23s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 42b2541 · merge 42b2541
centralus alert, e2e failed ★#7019 (merged) 3
e2e (1)
private ingress should not be reachable from outside the VNet, but connection succeeded
20:01:54 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:170]: private ingress should not be reachable from outside the VNet, but connection succeeded
Expected an error to have occurred.  Got:
    <nil>: ...
alert (2)
alert [svc] ServiceMemoryDrift fired
20:01:54 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-19T21:47:04Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j0061568-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-2zqr9", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-2zqr9 (namespace arobit) on cluster ci01-j0061568-svc has averaged 124.8% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
alert [svc] FrontendPathMedianLatency fired
20:01:54 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T21:05:45Z
  Ended:       2026-09-19T21:11:46Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0061568-svc", component="frontend", method="put", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j0061568-svc).
1h 36m 19s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 7a275cf · merge 7a275cf
canadacentral alert, e2e failed ★#7024 (merged) 4
e2e (3)
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
20:01:44 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000a06120>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wh7bztj94wh4" to finish creating: GET https://rp.j8800640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2",
      "name": "a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2",
      "status": "Failed",
      "startTime": "2026-09-19T20:34:43.428610597Z",
      "endTime": "2026-09-19T21:03:49.814289269Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000a06120>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-wh7bztj94wh4" to finish creating: GET https://rp.j8800640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2",
      "name": "a9bd4a4e-b6a1-4ac5-a23e-ee09a38de2f2",
      "status": "Failed",
      "startTime": "2026-09-19T20:34:43.428610597Z",
      "endTime": "2026-09-19T21:03:49.814289269Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 24m7s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
20:01:44 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000d24520>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-fgvlnn5kdj25" to finish creating: GET https://rp.j8800640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ec9c856-afdc-48ab-ad6a-4c93e8b9216a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ec9c856-afdc-48ab-ad6a-4c93e8b9216a",
      "name": "8ec9c856-afdc-48ab-ad6a-4c93e8b9216a",
      "status": "Failed",
      "startTime": "2026-09-19T20:35:08.292814934Z",
      "endTime": "2026-09-19T21:04:09.799274903Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 24m13s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000d24520>: 
    failed waiting for cluster="cilium-cluster" in resourcegroup="complex-cilium-kv-fgvlnn5kdj25" to finish creating: GET https://rp.j8800640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ec9c856-afdc-48ab-ad6a-4c93e8b9216a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/8ec9c856-afdc-48ab-ad6a-4c93e8b9216a",
      "name": "8ec9c856-afdc-48ab-ad6a-4c93e8b9216a",
      "status": "Failed",
      "startTime": "2026-09-19T20:35:08.292814934Z",
      "endTime": "2026-09-19T21:04:09.799274903Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 24m13s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
20:01:44 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0001da320>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-rpq79wd7j7md" to finish creating: GET https://rp.j8800640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6e591bf2-2064-49c7-857a-322459bd75aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6e591bf2-2064-49c7-857a-322459bd75aa",
      "name": "6e591bf2-2064-49c7-857a-322459bd75aa",
      "status": "Failed",
      "startTime": "2026-09-19T20:34:05.40086812Z",
      "endTime": "2026-09-19T21:03:09.741249314Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m39s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0001da320>: 
    failed to create HCP cluster private-kas: failed waiting for cluster="private-kas" in resourcegroup="private-kas-rpq79wd7j7md" to finish creating: GET https://rp.j8800640.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6e591bf2-2064-49c7-857a-322459bd75aa
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/6e591bf2-2064-49c7-857a-322459bd75aa",
      "name": "6e591bf2-2064-49c7-857a-322459bd75aa",
      "status": "Failed",
      "startTime": "2026-09-19T20:34:05.40086812Z",
      "endTime": "2026-09-19T21:03:09.741249314Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.22.14 is Partial (want Completed), started 22m39s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
alert (1)
alert [hcp] KubePodNotReady fired
20:01:44 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 3 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T20:45:34Z
  Ended:       2026-09-19T21:12:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8800640-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sua7p1uu2oiunlri0cqe315nk9q9n8f-n1o6p7z8w6z4r2q", pod="router-777b99f8d7-8drf8", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sua7p1uu2oiunlri0cqe315nk9q9n8f-n1o6p7z8w6z4r2q/router-777b99f8d7-8drf8 has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-19T20:46:34Z
  Ended:       2026-09-19T21:11:36Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8800640-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sua7fdtmm97d9evql41ln5ik252m94i-private-kas", pod="router-685c668c54-ftrhk", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sua7fdtmm97d9evql41ln5ik252m94i-private-kas/router-685c668c54-ftrhk has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-19T20:46:34Z
  Ended:       2026-09-19T21:12:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j8800640-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2sua7v07mnvun6l156puv4d57ktoj9ph-cilium-cluster", pod="router-9c9b7989-jmkt7", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2sua7v07mnvun6l156puv4d57ktoj9ph-cilium-cluster/router-9c9b7989-jmkt7 has been in a non-ready state for longer than 5 minutes.
1h 39m 15s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head d450510 · merge d450510
centralus alert failed ★#7021 (merged) 1
alert (1)
alert [svc] FrontendPathLatency fired
19:48:02 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathLatency does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T20:58:11Z
  Ended:       2026-09-19T21:04:12Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j8848768-svc", component="frontend", method="get", region="centralus", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} exceeded 1 second over the past 30 minutes (cluster ci01-j8848768-svc).
1h 58m 24s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 6cae5a1 · merge 6cae5a1
canadacentral alert failed ★#7023 (merged) 1
alert (1)
alert [svc] FrontendPathMedianLatency fired
19:47:19 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T21:05:08Z
  Ended:       2026-09-19T21:11:08Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j3425408-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j3425408-svc).
32m 30s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head d450510 · merge d450510
canadacentral provision failed ★#7021 (merged) 1
provision (1)
ERROR CODE: DeploymentFailed; detail code VaultAlreadyExists; detail message the vault name '<vault-name>' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link <url>
19:00:57 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/infra (stamp=1) · suite step graph
Show raw failure
time=2026-09-19T19:26:02.513Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=1 description="Step infra\n  Kind: ARM\n  Template: templates/mgmt-infra.bicep\n  Parameters: configurations/mgmt-infra.tmpl.bicepparam"
time=2026-09-19T19:26:03.032Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=1
time=2026-09-19T19:26:04.005Z level=DEBUG+3 msg="Deployment started" serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=1 deployment=6b08ae6c0b84b662e34009b99333555395423c7dc45a028e443e73278d202759 portal=https://ms.portal.azure.com/#view/Microsoft_Azure_Resources/DeploymentDetails.MenuView/~/overview/id/%2Fsubscriptions%2FXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX%2FresourceGroups%2Fhcp-underlay-ci01-j3811200-mgmt-1%2Fproviders%2FMicrosoft.Resources%2Fdeployments%2F6b08ae6c0b84b662e34009b99333555395423c7dc45a028e443e73278d202759
time=2026-09-19T19:28:34.855Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=infra stamp=1 err="stamp 1: failed to run ARM step: failed to poll deployment: failed to wait for deployment completion: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j3811200-mgmt-1/providers/Microsoft.Resources/deployments/6b08ae6c0b84b662e34009b99333555395423c7dc45a028e443e73278d202759/operationStatuses/08584117609220848049\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Conflict\",\n        \"message\": \"{\\r\\n  \\\"status\\\": \\\"Failed\\\",\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"ResourceDeploymentFailure\\\",\\r\\n    \\\"message\\\": \\\"The resource write operation failed to complete successfully, because it reached terminal provisioning state 'Failed'.\\\",\\r\\n    \\\"details\\\": [\\r\\n      {\\r\\n        \\\"code\\\": \\\"DeploymentFailed\\\",\\r\\n        \\\"target\\\": \\\"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/hcp-underlay-ci01-j3811200-mgmt-1/providers/Microsoft.Resources/deployments/ah-ci01-me-j3811200-1\\\",\\r\\n        \\\"message\\\": \\\"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\\\",\\r\\n        \\\"details\\\": [\\r\\n          {\\r\\n            \\\"code\\\": \\\"VaultAlreadyExists\\\",\\r\\n            \\\"message\\\": \\\"The vault name 'ah-ci01-me-j3811200-1' is already in use. Vault names are globally unique so it is possible that the name is already taken. If you are sure that the vault name was not taken then it is possible that a vault with the same name was recently deleted but not purged after being placed in a recoverable state. If the vault is in a recoverable state then the vault will need to be purged before reusing the name. For more information about VaultAlreadyExists, soft delete and purging a vault follow this link https://go.microsoft.com/fwlink/?linkid=2147740.\\\"\\r\\n          }\\r\\n        ]\\r\\n      }\\r\\n    ]\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n"
1h 52m 43s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 7a275cf · merge 7a275cf
centralus alert, e2e failed ★#7024 (merged) 3
e2e (2)
ERROR CODE: Canceled; detail message This operation was superseded by another; provider Microsoft.RedHatOpenShift
17:39:51 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-l9txx6 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000d2ad80>: 
    failed to create HCP cluster np-version-upgrade-cluster-l9txx6: failed waiting for cluster="np-version-upgrade-cluster-l9txx6" in resourcegroup="rg-np-version-upgrade-l9txx6-mwmzt2xrcxvt" to finish creating: GET https://rp.j2215680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ca9f3ef6-29fc-49b7-a060-580e48d94813
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: Canceled
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ca9f3ef6-29fc-49b7-a060-580e48d94813",
      "name": "ca9f3ef6-29fc-49b7-a060-580e48d94813",
      "status": "Canceled",
      "startTime": "2026-09-19T18:21:32.692382703Z",
      "endTime": "2026-09-19T18:21:36.063277066Z",
      "error": {
        "code": "Canceled",
        "message": "This operation was superseded by another"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-l9txx6 with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000d2ad80>: 
    failed to create HCP cluster np-version-upgrade-cluster-l9txx6: failed waiting for cluster="np-version-upgrade-cluster-l9txx6" in resourcegroup="rg-np-version-upgrade-l9txx6-mwmzt2xrcxvt" to finish creating: GET https://rp.j2215680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ca9f3ef6-29fc-49b7-a060-580e48d94813
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: Canceled
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/ca9f3ef6-29fc-49b7-a060-580e48d94813",
      "name": "ca9f3ef6-29fc-49b7-a060-580e48d94813",
      "status": "Canceled",
      "startTime": "2026-09-19T18:21:32.692382703Z",
      "endTime": "2026-09-19T18:21:36.063277066Z",
      "error": {
        "code": "Canceled",
        "message": "This operation was superseded by another"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: Conflict; detail message Cannot update resource while resource is "accepted"; provider Microsoft.RedHatOpenShift
17:39:51 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-l9txx6 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc00167c880>: 
    failed to create HCP cluster np-version-upgrade-cluster-l9txx6: failed starting cluster creation "np-version-upgrade-cluster-l9txx6" in resourcegroup="rg-np-version-upgrade-l9txx6-mwmzt2xrcxvt": PUT https://rp.j2215680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-l9txx6-mwmzt2xrcxvt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-l9txx6
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"accepted\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-l9txx6-mwmzt2xrcxvt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-l9txx6"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-l9txx6 with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc00167c880>: 
    failed to create HCP cluster np-version-upgrade-cluster-l9txx6: failed starting cluster creation "np-version-upgrade-cluster-l9txx6" in resourcegroup="rg-np-version-upgrade-l9txx6-mwmzt2xrcxvt": PUT https://rp.j2215680.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-l9txx6-mwmzt2xrcxvt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-l9txx6
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: Conflict
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "Conflict",
        "message": "Cannot update resource while resource is \"accepted\"",
        "target": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-l9txx6-mwmzt2xrcxvt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-l9txx6"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
alert (1)
alert [hcp] KubePodNotReady fired
17:39:51 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 7 time(s) (1 unknown, 6 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T18:54:13Z
  Ended:       2026-09-19T19:00:14Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j2215680-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su88o47irf75plc21jkh6milgqqj80l-s6c0t0c0k6r3x8q", pod="router-6b99f777f5-vr4nm", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su88o47irf75plc21jkh6milgqqj80l-s6c0t0c0k6r3x8q/router-6b99f777f5-vr4nm has been in a non-ready state for longer than 5 minutes.
2h 13m 0s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 42b2541 · merge 42b2541
centralus alert failed ★#7019 (merged) 2
alert (2)
alert [svc] ServiceMemoryDrift fired
17:17:27 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-19T19:20:34Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j1596032-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-zb5zr", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-zb5zr (namespace arobit) on cluster ci01-j1596032-svc has averaged 124.8% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
alert [hcp] KubePodNotReady fired
17:17:27 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 27 time(s) (15 unknown, 12 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:26:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="cluster-node-tuning-operator-786575b7cc-rqwdt", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/cluster-node-tuning-operator-786575b7cc-rqwdt has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:27:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="cluster-image-registry-operator-7d5c8cfb-2qs65", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/cluster-image-registry-operator-7d5c8cfb-2qs65 has been in a non-ready state for longer than 5 minutes.
Firing 3:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:27:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="csi-snapshot-controller-operator-5bf7f97b88-vcs98", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/csi-snapshot-controller-operator-5bf7f97b88-vcs98 has been in a non-ready state for longer than 5 minutes.
Firing 4:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:26:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82us6cpqcoa1ofkcab6a61g68g84r-private-ingress", pod="router-58d6c54947-vtg65", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82us6cpqcoa1ofkcab6a61g68g84r-private-ingress/router-58d6c54947-vtg65 has been in a non-ready state for longer than 5 minutes.
Firing 5:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:26:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="openshift-controller-manager-666cf5fc85-56gwb", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/openshift-controller-manager-666cf5fc85-56gwb has been in a non-ready state for longer than 5 minutes.
Firing 6:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:27:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su8434ht5kskjt0q79ufdqc9r17debr-agg-cluster", pod="etcd-0", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su8434ht5kskjt0q79ufdqc9r17debr-agg-cluster/etcd-0 has been in a non-ready state for longer than 5 minutes.
Firing 7:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:26:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="cluster-policy-controller-65b5b47d99-l5lwt", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/cluster-policy-controller-65b5b47d99-l5lwt has been in a non-ready state for longer than 5 minutes.
Firing 8:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:26:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="ingress-operator-ffb8cff7-lfknp", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/ingress-operator-ffb8cff7-lfknp has been in a non-ready state for longer than 5 minutes.
Firing 9:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:26:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="openshift-route-controller-manager-b76f84b4d-qq97g", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/openshift-route-controller-manager-b76f84b4d-qq97g has been in a non-ready state for longer than 5 minutes.
Firing 10:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:26:24Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="cluster-version-operator-85487bbffd-ldcp7", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/cluster-version-operator-85487bbffd-ldcp7 has been in a non-ready state for longer than 5 minutes.
Firing 11:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:27:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="cluster-network-operator-9498c74fd-sn2n9", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/cluster-network-operator-9498c74fd-sn2n9 has been in a non-ready state for longer than 5 minutes.
Firing 12:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:26:23Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="control-plane-pki-operator-9cf9f89cb-n5sjj", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/control-plane-pki-operator-9cf9f89cb-n5sjj has been in a non-ready state for longer than 5 minutes.
Firing 13:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:27:33Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="dns-operator-c98595c87-q5r9m", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/dns-operator-c98595c87-q5r9m has been in a non-ready state for longer than 5 minutes.
Firing 14:
  State:       Resolved
  Started:     2026-09-19T18:20:23Z
  Ended:       2026-09-19T18:27:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="cluster-storage-operator-565c99cf9-2wfg2", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/cluster-storage-operator-565c99cf9-2wfg2 has been in a non-ready state for longer than 5 minutes.
Firing 15:
  State:       Resolved
  Started:     2026-09-19T18:21:24Z
  Ended:       2026-09-19T18:27:34Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1596032-mgmt-1", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a", pod="certified-operators-catalog-65bc785bb-lxzgs", region="centralus", severity="warning"
  Description: Pod ocm-arohcpci01-2su82uvdus7kmj5gr0r2eh33bf7c1edf-u8z7v1e7s1c5h5a/certified-operators-catalog-65bc785bb-lxzgs has been in a non-ready state for longer than 5 minutes.
2h 11m 13s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head f177240 · merge f177240
westus3 n/a passed #7017 (merged) 0 n/a
47m 14s pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 17c08cd · merge 6cae5a1
canadacentral provision failed ⚠#7023 (merged) 1
provision (1)
failed to roll out Helm release: resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 0/15 context deadline exceeded
12:50:02 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Management.Infra/management/kubelet-ds (stamp=1) · suite step graph
Show raw failure
time=2026-09-19T13:26:44.501Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1 description="Step kubelet-ds\n Kind: Helm\n"
time=2026-09-19T13:26:44.949Z level=INFO msg="Resolved input values." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1 values=map[enabled:true]
time=2026-09-19T13:26:44.949Z level=INFO msg="Applying namespaces." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1
time=2026-09-19T13:26:45.277Z level=INFO msg="Namespace applied successfully." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1 namespace=kubelet-ds
time=2026-09-19T13:26:45.468Z level=INFO msg="Rolling out Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1 dryRun=false
time=2026-09-19T13:26:45.468Z level=INFO msg="Searching for release history..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1
time=2026-09-19T13:26:45.524Z level=INFO msg="No release history found, running the first release..." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1
time=2026-09-19T13:31:46.698Z level=ERROR msg="Failed to roll out the Helm release." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1 err="resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 0/15\ncontext deadline exceeded"
time=2026-09-19T13:31:46.698Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Management.Infra resourceGroup=management step=kubelet-ds stamp=1 err="stamp 1: error running Helm release deployment Step, failed to deploy helm release: failed to roll out Helm release: resource DaemonSet/kube-system/set-kubelet-parameters-for-scale not ready. status: InProgress, message: Available: 0/15\ncontext deadline exceeded"
2h 17m 45s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head f177240 · merge f177240
westus3 e2e failed ★#7017 (merged) 1
e2e (1)
ERROR CODE: InternalServerError; detail message cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster still exists; [clusterServiceStatus] state is "uninstalling"; [descendantResources] remaining resources: serviceProviderClusters; provider Microsoft.RedHatOpenShift
12:45:48 UTC · failed at e2e · test Customer should be able to rotate KMS key for a cluster with version >= 4.22 · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000fce138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-xvkwtqlz7wnc" to finish deleting: GET https://rp.j8827264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b599b74-9276-4eb3-b0df-f51605e1e94b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b599b74-9276-4eb3-b0df-f51605e1e94b",
      "name": "7b599b74-9276-4eb3-b0df-f51605e1e94b",
      "status": "Failed",
      "startTime": "2026-09-19T14:20:22.933309802Z",
      "endTime": "2026-09-19T14:44:28.067256394Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2su42fk9ipo2ddicrn6ki899u5dtl5la still exists (deletion dispatched at 2026-09-19T14:20:25Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/util/framework/per_test_framework.go:295]: Unexpected error:
    <*errors.joinError | 0xc000fce138>: 
    failed to cleanup resource group: at least one hcp cluster failed to delete: failed waiting for hcpCluster="kms-key-rotate-422" in resourcegroup="kms-key-rotate-xvkwtqlz7wnc" to finish deleting: GET https://rp.j8827264.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b599b74-9276-4eb3-b0df-f51605e1e94b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/7b599b74-9276-4eb3-b0df-f51605e1e94b",
      "name": "7b599b74-9276-4eb3-b0df-f51605e1e94b",
      "status": "Failed",
      "startTime": "2026-09-19T14:20:22.933309802Z",
      "endTime": "2026-09-19T14:44:28.067256394Z",
      "error": {
        "code": "InternalServerError",
        "message": "cluster deletion did not complete before the deadline; [clusterServiceDeletion] ClusterService cluster /api/aro_hcp/v1alpha1/clusters/2su42fk9ipo2ddicrn6ki899u5dtl5la still exists (deletion dispatched at 2026-09-19T14:20:25Z); [clusterServiceStatus] ClusterService state is \"uninstalling\"; [descendantResources] remaining resources: 1 microsoft.redhatopenshift/hcpopenshiftclusters/serviceProviderClusters"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
1h 41m 47s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 323e517 · merge 323e517
westus3 n/a passed #7013 (merged) 0 n/a
1h 58m 7s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head d450510 · merge d450510
westus3 alert, e2e failed ★#7021 (merged) 2
e2e (1)
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
12:07:24 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:738]: failed to create nodepool npdg-422-420
Unexpected error:
    <*fmt.wrapError | 0xc000cd7540>: 
    failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-zrpb7z" in resourcegroup="rg-np-dg-minor-zrpb7z-2p25vj8zqmn4" to finish creating: GET https://rp.j0702464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bec0f664-bd6a-4d78-97ad-4416d4ab5f15
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bec0f664-bd6a-4d78-97ad-4416d4ab5f15",
      "name": "bec0f664-bd6a-4d78-97ad-4416d4ab5f15",
      "status": "Failed",
      "startTime": "2026-09-19T13:03:18.457829152Z",
      "endTime": "2026-09-19T13:22:23.311283757Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:738]: failed to create nodepool npdg-422-420
Unexpected error:
    <*fmt.wrapError | 0xc000cd7540>: 
    failed to create NodePool npdg-422-420: failed waiting for nodepool="npdg-422-420" for cluster "np-dg-minor-zrpb7z" in resourcegroup="rg-np-dg-minor-zrpb7z-2p25vj8zqmn4" to finish creating: GET https://rp.j0702464.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bec0f664-bd6a-4d78-97ad-4416d4ab5f15
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/westus3/hcpOperationStatuses/bec0f664-bd6a-4d78-97ad-4416d4ab5f15",
      "name": "bec0f664-bd6a-4d78-97ad-4416d4ab5f15",
      "status": "Failed",
      "startTime": "2026-09-19T13:03:18.457829152Z",
      "endTime": "2026-09-19T13:22:23.311283757Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
alert (1)
alert [svc] FrontendPathMedianLatency fired
12:07:24 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T13:13:57Z
  Ended:       2026-09-19T13:19:59Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j0702464-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: The median (p50) of frontend request latency for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} has exceeded 250ms over the past 30 minutes (cluster ci01-j0702464-svc).
1h 55m 37s pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 8c7e1e0 · merge 244d0c1
canadacentral e2e failed ★#4878 (merged) 1
e2e (1)
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
10:22:26 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc001364120>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-49gn4qjgmx5r" to finish creating: GET https://rp.j7481216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d405444-e5c6-413b-8e97-5fb154fce110
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d405444-e5c6-413b-8e97-5fb154fce110",
      "name": "5d405444-e5c6-413b-8e97-5fb154fce110",
      "status": "Failed",
      "startTime": "2026-09-19T11:23:33.676780621Z",
      "endTime": "2026-09-19T11:42:34.970958081Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc001364120>: 
    failed to create NodePool np-delete-main: failed waiting for nodepool="np-delete-main" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-49gn4qjgmx5r" to finish creating: GET https://rp.j7481216.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d405444-e5c6-413b-8e97-5fb154fce110
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/5d405444-e5c6-413b-8e97-5fb154fce110",
      "name": "5d405444-e5c6-413b-8e97-5fb154fce110",
      "status": "Failed",
      "startTime": "2026-09-19T11:23:33.676780621Z",
      "endTime": "2026-09-19T11:42:34.970958081Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
2h 8m 55s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 69ee8b9
centralus n/a passed #6993 (open) 0 n/a
2h 12m 5s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 323e517 · merge 323e517
canadacentral alert failed ★#7013 (merged) 1
alert (1)
alert [svc] BackendControllerQueueDepthHigh fired
10:21:41 UTC · failed at alert · test [aro-hcp-observability] [svc] alert BackendControllerQueueDepthHigh does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T11:30:53Z
  Ended:       2026-09-19T11:44:53Z
  Severity:    Sev3
  Labels:      alertname="BackendControllerQueueDepthHigh", cluster="ci01-j4648192-svc", component="backend", name="clusterresources", region="canadacentral", severity="warning"
  Description: Backend controller workqueue clusterresources has had a depth > 10 for more than 15 minutes, indicating work is accumulating faster than it can be processed.
2h 2m 54s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 9ef5982
westus3 alert failed ★#6925 (open) 3
alert (3)
alert [svc] FrontendPathLatency fired
10:21:02 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathLatency does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T11:44:47Z
  Ended:       2026-09-19T12:09:48Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathLatency", cluster="ci01-j9539072-svc", component="frontend", method="get", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname}", severity="warning"
  Description: More than 5% of frontend requests for get /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}/nodepools/{nodepoolname} exceeded 1 second over the past 30 minutes (cluster ci01-j9539072-svc).
alert [svc] ServiceMemoryDrift fired
10:21:02 UTC · failed at alert · test [aro-hcp-observability] [svc] alert ServiceMemoryDrift does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Fired (not resolved)
  Started:     2026-09-19T12:10:12Z
  Severity:    Sev3
  Labels:      alertname="ServiceMemoryDrift", cluster="ci01-j9539072-svc", component="service-memory", container="fluentbit", namespace="arobit", pod="arobit-forwarder-6qrsm", severity="warning", team="hcp-sl"
  Description: Container fluentbit in pod arobit-forwarder-6qrsm (namespace arobit) on cluster ci01-j9539072-svc has averaged 124% of its memory request over the last 30 minutes.
This may indicate a memory leak or workload growth that requires right-sizing the request in config.yaml.
alert [svc] FrontendPathMedianLatency fired
10:21:02 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T11:41:47Z
  Ended:       2026-09-19T11:47:47Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j9539072-svc", component="frontend", method="put", region="westus3", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j9539072-svc).
1h 56m 40s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head b9f8fb1 · merge b9f8fb1
westus3 n/a passed #7010 (merged) 0 n/a
1h 58m 17s pull-ci-Azure-ARO-HCP-main-e2e-parallel
merged PR
head 12d6469 · merge d450510
canadacentral alert, e2e failed ★#7021 (merged) 5
e2e (2)
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster control plane version not yet completed: version <version> is Partial (want Completed), started <duration> ago; hosted cluster degraded: UnavailableReplicas: [router]; provider Microsoft.RedHatOpenShift
02:41:11 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc0008d42e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-g25b4wmpwjv7" to finish creating: GET https://rp.j1862272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf395da6-922a-48c5-9fac-e843414308f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf395da6-922a-48c5-9fac-e843414308f1",
      "name": "bf395da6-922a-48c5-9fac-e843414308f1",
      "status": "Failed",
      "startTime": "2026-09-19T03:27:32.631091464Z",
      "endTime": "2026-09-19T03:56:32.903456595Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 22m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc0008d42e0>: 
    failed waiting for cluster="fips-enabled-cluster" in resourcegroup="fips-enabled-g25b4wmpwjv7" to finish creating: GET https://rp.j1862272.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf395da6-922a-48c5-9fac-e843414308f1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/canadacentral/hcpOperationStatuses/bf395da6-922a-48c5-9fac-e843414308f1",
      "name": "bf395da6-922a-48c5-9fac-e843414308f1",
      "status": "Failed",
      "startTime": "2026-09-19T03:27:32.631091464Z",
      "endTime": "2026-09-19T03:56:32.903456595Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster control plane version not yet completed: version 4.21.34 is Partial (want Completed), started 22m24s ago; hosted cluster degraded: UnavailableReplicas: router deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
cluster entered terminal Failed state after role assignment deployment
02:41:11 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:227]: Timed out after 2580.001s.
cluster should eventually succeed after role assignments are created
The function passed to Eventually failed at /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:223 with:
cluster entered terminal Failed state after role assignment deployment
Expected
    <armredhatopenshifthcp.ProvisioningState>: ...
not to equal
    <armredhatopenshifthcp.ProvisioningState>: ...
alert (3)
alert [hcp] KubePodNotReady fired
02:41:11 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodNotReady does not fire · suite aro-hcp-tests
Show raw failure
alert fired 4 time(s) (2 unknown, 2 known)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T03:38:09Z
  Ended:       2026-09-19T04:25:09Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1862272-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2str5vc78gg1rc75eamfsefvm7hjps76-p8y4e2f9o6p1q5z", pod="router-84cc47c7c9-zk9pv", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2str5vc78gg1rc75eamfsefvm7hjps76-p8y4e2f9o6p1q5z/router-84cc47c7c9-zk9pv has been in a non-ready state for longer than 5 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-19T03:41:09Z
  Ended:       2026-09-19T04:09:09Z
  Severity:    Sev3
  Labels:      alertname="KubePodNotReady", cluster="ci01-j1862272-mgmt-2", component="kubernetes-infrastructure", namespace="ocm-arohcpci01-2str69dfnmshvfdpbh2s8nosldfesjea-b4z4j4n6g4b4k4k", pod="router-bf8f56fc6-l2mdj", region="canadacentral", severity="warning"
  Description: Pod ocm-arohcpci01-2str69dfnmshvfdpbh2s8nosldfesjea-b4z4j4n6g4b4k4k/router-bf8f56fc6-l2mdj has been in a non-ready state for longer than 5 minutes.
alert [svc] FrontendPathMedianLatency fired
02:41:11 UTC · failed at alert · test [aro-hcp-observability] [svc] alert FrontendPathMedianLatency does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T03:59:08Z
  Ended:       2026-09-19T04:05:09Z
  Severity:    Sev3
  Labels:      alertname="FrontendPathMedianLatency", cluster="ci01-j1862272-svc", component="frontend", method="put", region="canadacentral", route="/subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename}", severity="warning"
  Description: The median (p50) of frontend request latency for put /subscriptions/{subscriptionid}/resourcegroups/{resourcegroupname}/providers/microsoft.redhatopenshift/hcpopenshiftclusters/{resourcename} has exceeded 250ms over the past 30 minutes (cluster ci01-j1862272-svc).
alert [hcp] KubeDeploymentReplicasMismatch fired
02:41:11 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T04:03:08Z
  Ended:       2026-09-19T04:21:10Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j1862272-mgmt-2", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="router", endpoint="http", environment="ci01", instance="10.128.64.253:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j1862272/providers/microsoft.monitor/accounts/hcps-j1862272", namespace="ocm-arohcpci01-2str5vc78gg1rc75eamfsefvm7hjps76-p8y4e2f9o6p1q5z", pod="arohcp-monitor-kube-state-metrics-7955558448-8g9rn", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="canadacentral", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2str5vc78gg1rc75eamfsefvm7hjps76-p8y4e2f9o6p1q5z/router has not matched the expected number of replicas for longer than 30 minutes.
1h 44m 2s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 2069fe0 · merge 2069fe0
westus3 n/a passed #7012 (merged) 0 n/a
23m 46s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head 14b5322
centralus provision failed ⚠#7020 (open) 1
provision (1)
ERROR CODE: InvalidTemplateDeployment; provider Microsoft.Authorization
00:40:35 UTC · failed at provision · test Run pipeline step Microsoft.Azure.ARO.HCP.Service.Infra/regional/adminapi-alert-rule-role · suite step graph
Show raw failure
time=2026-09-19T00:59:12.578Z level=INFO msg="Running step." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=regional step=adminapi-alert-rule-role description="Step adminapi-alert-rule-role\n  Kind: ARM\n  Template: templates/adminapi-alert-processing-rule-role.bicep\n  Parameters: configurations/adminapi-alert-processing-rule-role.tmpl.bicepparam"
time=2026-09-19T00:59:12.962Z level=DEBUG+2 msg="Starting ARM deployment" serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=regional step=adminapi-alert-rule-role
time=2026-09-19T00:59:13.303Z level=ERROR msg="Step errored." serviceGroup=Microsoft.Azure.ARO.HCP.Service.Infra resourceGroup=regional step=adminapi-alert-rule-role err="failed to run ARM step: failed to create deployment: PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Resources/deployments/73540b7e4361f329d8cc7347416ba379091383d8ff0e1f7fb10b7cc1abb08e96\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplateDeployment\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidTemplateDeployment\",\n    \"message\": \"The template deployment failed with error: 'Authorization failed for template resource '964da0aa-1cca-5874-beb3-3734babb6fde' of type 'Microsoft.Authorization/roleDefinitions'. The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id 'c209f8df-52ae-48fb-98ea-380f58b04652' does not have permission to perform action 'Microsoft.Authorization/roleDefinitions/write' at scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.Authorization/roleDefinitions/964da0aa-1cca-5874-beb3-3734babb6fde'.'.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
1h 53m 19s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head 42b2541 · merge 42b2541
centralus alert, e2e failed ★#7019 (merged) 4
e2e (1)
ERROR CODE: InternalServerError; detail message [hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: [capi-provider]; provider Microsoft.RedHatOpenShift
00:27:13 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite rp-api-compat-all/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc00094e420>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-dk96slfsvrkf" to finish creating: GET https://rp.j8467840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3ef44c4c-4b9e-402d-8da6-a1c2254a2bba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3ef44c4c-4b9e-402d-8da6-a1c2254a2bba",
      "name": "3ef44c4c-4b9e-402d-8da6-a1c2254a2bba",
      "status": "Failed",
      "startTime": "2026-09-19T01:06:55.31906784Z",
      "endTime": "2026-09-19T01:36:02.073820212Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc00094e420>: 
    failed to create HCP cluster ingress-cert-le: failed waiting for cluster="ingress-cert-le" in resourcegroup="ingress-cert-le-dk96slfsvrkf" to finish creating: GET https://rp.j8467840.hcpsvc.osadev.cloud/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3ef44c4c-4b9e-402d-8da6-a1c2254a2bba
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/centralus/hcpOperationStatuses/3ef44c4c-4b9e-402d-8da6-a1c2254a2bba",
      "name": "3ef44c4c-4b9e-402d-8da6-a1c2254a2bba",
      "status": "Failed",
      "startTime": "2026-09-19T01:06:55.31906784Z",
      "endTime": "2026-09-19T01:36:02.073820212Z",
      "error": {
        "code": "InternalServerError",
        "message": "[hypershiftHostedCluster] hosted cluster is not available: KubeconfigWaitingForCreate: Waiting for hosted control plane kubeconfig to be created; hosted cluster degraded: UnavailableReplicas: capi-provider deployment has 1 unavailable replicas"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
alert (3)
alert [hcp] KubeStatefulSetReplicasMismatch fired
00:27:13 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeStatefulSetReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T01:28:47Z
  Ended:       2026-09-19T01:54:50Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8467840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8467840/providers/microsoft.monitor/accounts/hcps-j8467840", namespace="ocm-arohcpci01-2stp4bsnt5nt46fh2hgau5todka5ufn5-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-6486647766-4mw7w", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2stp4bsnt5nt46fh2hgau5todka5ufn5-ingress-cert-le/etcd has not matched the expected number of replicas for longer than 15 minutes.
Firing 2:
  State:       Resolved
  Started:     2026-09-19T01:29:59Z
  Ended:       2026-09-19T01:56:46Z
  Severity:    Sev3
  Labels:      alertname="KubeStatefulSetReplicasMismatch", cluster="ci01-j8467840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", endpoint="http", environment="ci01", instance="10.128.65.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8467840/providers/microsoft.monitor/accounts/hcps-j8467840", namespace="ocm-arohcpci01-2stp4bsnt5nt46fh2hgau5todka5ufn5-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-6486647766-4mw7w", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", statefulset="etcd"
  Description: StatefulSet ocm-arohcpci01-2stp4bsnt5nt46fh2hgau5todka5ufn5-ingress-cert-le/etcd has not matched the expected number of replicas for longer than 15 minutes.
alert [hcp] KubeDeploymentReplicasMismatch fired
00:27:13 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubeDeploymentReplicasMismatch does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T01:43:47Z
  Ended:       2026-09-19T01:58:47Z
  Severity:    Sev3
  Labels:      alertname="KubeDeploymentReplicasMismatch", cluster="ci01-j8467840-mgmt-1", component="kubernetes-infrastructure", container="kube-state-metrics", deployment="capi-provider", endpoint="http", environment="ci01", instance="10.128.65.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8467840/providers/microsoft.monitor/accounts/hcps-j8467840", namespace="ocm-arohcpci01-2stp4bsnt5nt46fh2hgau5todka5ufn5-ingress-cert-le", pod="arohcp-monitor-kube-state-metrics-6486647766-4mw7w", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-0", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning"
  Description: Deployment ocm-arohcpci01-2stp4bsnt5nt46fh2hgau5todka5ufn5-ingress-cert-le/capi-provider has not matched the expected number of replicas for longer than 30 minutes.
alert [hcp] KubePodCrashLooping fired
00:27:13 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert KubePodCrashLooping does not fire · suite aro-hcp-tests
Show raw failure
alert fired 1 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T01:30:45Z
  Ended:       2026-09-19T01:58:45Z
  Severity:    Sev3
  Labels:      alertname="KubePodCrashLooping", cluster="ci01-j8467840-mgmt-1", component="kubernetes-infrastructure", container="etcd", endpoint="http", environment="ci01", instance="10.128.65.52:8080", job="kube-state-metrics", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8467840/providers/microsoft.monitor/accounts/hcps-j8467840", namespace="ocm-arohcpci01-2stp4bsnt5nt46fh2hgau5todka5ufn5-ingress-cert-le", pod="etcd-0", prometheus="prometheus/prometheus", prometheus_replica="prom-agent-prometheus-1", reason="CrashLoopBackOff", region="centralus", service="arohcp-monitor-kube-state-metrics", severity="warning", uid="daf93e4d-d43f-49cd-bbc8-4620650d35fc"
  Description: Pod ocm-arohcpci01-2stp4bsnt5nt46fh2hgau5todka5ufn5-ingress-cert-le/etcd-0 (etcd) is in waiting state (reason: "CrashLoopBackOff").
1h 51m 51s pull-ci-Azure-ARO-HCP-main-e2e-parallel
head f72c915
westus3 alert failed ⚠#6359 (open) 2
alert (2)
alert [hcp] userJourneySwiftRouterAvailability1h5m fired
00:25:15 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert userJourneySwiftRouterAvailability1h5m does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T01:16:42Z
  Ended:       2026-09-19T01:52:41Z
  Severity:    Sev3
  Labels:      alertname="userJourneySwiftRouterAvailability1h5m", burn_rate_tier="fast", cluster="ci01-j8883328-mgmt-1", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8883328/providers/microsoft.monitor/accounts/hcps-j8883328", region="westus3", severity="3", short_window="5m"
  Description: SWIFT router-capacity availability is consuming its error budget at the fast burn rate.
Firing 2:
  State:       Resolved
  Started:     2026-09-19T01:38:43Z
  Ended:       2026-09-19T01:47:43Z
  Severity:    Sev3
  Labels:      alertname="userJourneySwiftRouterAvailability1h5m", burn_rate_tier="fast", cluster="ci01-j8883328-mgmt-2", component="slo", long_window="1h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8883328/providers/microsoft.monitor/accounts/hcps-j8883328", region="westus3", severity="3", short_window="5m"
  Description: SWIFT router-capacity availability is consuming its error budget at the fast burn rate.
alert [hcp] userJourneySwiftRouterAvailability6h30m fired
00:25:15 UTC · failed at alert · test [aro-hcp-observability] [hcp] alert userJourneySwiftRouterAvailability6h30m does not fire · suite aro-hcp-tests
Show raw failure
alert fired 2 time(s)
Firing 1:
  State:       Resolved
  Started:     2026-09-19T01:39:45Z
  Ended:       2026-09-19T01:51:43Z
  Severity:    Sev3
  Labels:      alertname="userJourneySwiftRouterAvailability6h30m", burn_rate_tier="medium", cluster="ci01-j8883328-mgmt-2", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8883328/providers/microsoft.monitor/accounts/hcps-j8883328", region="westus3", severity="3", short_window="30m"
  Description: SWIFT router-capacity availability is consuming its error budget at the medium burn rate.
Firing 2:
  State:       Resolved
  Started:     2026-09-19T01:41:44Z
  Ended:       2026-09-19T01:56:44Z
  Severity:    Sev3
  Labels:      alertname="userJourneySwiftRouterAvailability6h30m", burn_rate_tier="medium", cluster="ci01-j8883328-mgmt-1", component="slo", long_window="6h", microsoft.amwresourceid="/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/hcp-underlay-ci01-j8883328/providers/microsoft.monitor/accounts/hcps-j8883328", region="westus3", severity="3", short_window="30m"
  Description: SWIFT router-capacity availability is consuming its error budget at the medium burn rate.
2h 6m 40s pull-ci-Azure-ARO-HCP-main-e2e-parallel
post-goodmerged PR
head bf8010e · merge bf8010e
westus3 n/a passed #6639 (merged) 0 n/a

Environment: INT

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
2h 23m 2s branch-ci-Azure-ARO-HCP-main-e2e-integration-e2e-parallel
post-goodmerged PR
No additional run metadata captured
westus3 n/a passed n/a 0 n/a

Environment: PROD

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
2h 24m 53s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
uksouth e2e failed n/a 12
e2e (12)
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0015bd1a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8c9pqlg82rcv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:193]: failed to update HCP cluster tags via PATCH
Unexpected error:
    <*exported.ResponseError | 0xc0015bd1a0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8c9pqlg82rcv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
23:00:33 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc000ac15d8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-16-151028, targetMinor=5.1.0-0.nightly-multi-2026-09-15-135250) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-15-135250"
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:176]: Timed out after 2700.002s.
Expected success, but got an error:
    <*errors.joinError | 0xc000ac15d8>: 
    VerifyKubeAPIServerServerVersionUpgraded failed: kube-apiserver ServerVersion not updated (unchanged from pre-upgrade)
    VerifyHostedControlPlaneYStreamUpgrade(previousMinor=5.0.0-0.nightly-multi-2026-09-16-151028, targetMinor=5.1.0-0.nightly-multi-2026-09-15-135250) failed: clusterversion status.history has no version in target minor "5.1.0-0.nightly-multi-2026-09-15-135250"
    ...
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc00146b710>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-ghsbrb2bjbhl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:175]: failed to update cluster with second ImageDigestMirror set
Unexpected error:
    <*exported.ResponseError | 0xc00146b710>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-ghsbrb2bjbhl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc001f2b5f0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-grfq9zcv7hdn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:274]: failed to begin cluster "agg-cluster" authorized CIDR update
Unexpected error:
    <*exported.ResponseError | 0xc001f2b5f0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-grfq9zcv7hdn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc00175eea0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-cj457ljpp84w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:408]: failed to begin updating cluster "cidr-connectivity-test" authorized CIDRs
Unexpected error:
    <*exported.ResponseError | 0xc00175eea0>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-cj457ljpp84w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc0008d4380>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-7s7p96" in resourcegroup="rg-np-version-upgrade-7s7p96-pqsvgvw7tccb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7s7p96-pqsvgvw7tccb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-7s7p96/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:165]: failed to create node pool npupgrade-4-20 with version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc0008d4380>: 
    failed to create NodePool npupgrade-4-20: failed starting nodepool creation "npupgrade-4-20" for cluster "np-version-upgrade-cluster-7s7p96" in resourcegroup="rg-np-version-upgrade-7s7p96-pqsvgvw7tccb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-7s7p96-pqsvgvw7tccb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-7s7p96/nodePools/npupgrade-4-20
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InternalServerError; detail message node pool creation did not complete before the deadline; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc000a35460>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-zs4kjp29q62s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d09b8c89-85cf-4c6e-a3fe-60bf6886ed15
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d09b8c89-85cf-4c6e-a3fe-60bf6886ed15",
      "name": "d09b8c89-85cf-4c6e-a3fe-60bf6886ed15",
      "status": "Failed",
      "startTime": "2026-09-19T23:48:22.46928069Z",
      "endTime": "2026-09-20T00:07:28.436280978Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:123]: failed to create node pool ea-np-1
Unexpected error:
    <*fmt.wrapError | 0xc000a35460>: 
    failed to create NodePool ea-np-1: failed waiting for nodepool="ea-np-1" for cluster "ea-cluster" in resourcegroup="external-auth-cluster-zs4kjp29q62s" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d09b8c89-85cf-4c6e-a3fe-60bf6886ed15
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d09b8c89-85cf-4c6e-a3fe-60bf6886ed15",
      "name": "d09b8c89-85cf-4c6e-a3fe-60bf6886ed15",
      "status": "Failed",
      "startTime": "2026-09-19T23:48:22.46928069Z",
      "endTime": "2026-09-20T00:07:28.436280978Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.21": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc00297dbc0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-fsdprlzmvbsr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:173]: failed to update cluster "autoscaling-hcp-cluster" autoscaling configuration
Unexpected error:
    <*exported.ResponseError | 0xc00297dbc0>: 
    PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-fsdprlzmvbsr/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.21\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc0013a9700>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-t2vqgb" resourcegroup="rg-np-dg-minor-t2vqgb-v48wcn4f6v4f": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-t2vqgb-v48wcn4f6v4f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-t2vqgb/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:766]: failed to update nodepool npdg-422-420 to downgrade target 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc0013a9700>: 
    failed to start nodepool "npdg-422-420" update in cluster "np-dg-minor-t2vqgb" resourcegroup="rg-np-dg-minor-t2vqgb-v48wcn4f6v4f": PATCH https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-t2vqgb-v48wcn4f6v4f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-t2vqgb/nodePools/npdg-422-420
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "4.22": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is "nightly"; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00106b280>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-mbf8fbn585g6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-mbf8fbn585g6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc00106b280>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-mbf8fbn585g6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-mbf8fbn585g6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.22\": must be specified as MAJOR.MINOR.PATCH (optionally with a pre-release, e.g. a nightly build suffix) when channelGroup is \"nightly\"",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc00093ba00>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-w54s7z" in resourcegroup="rg-np-skip-minor-w54s7z-sj4xc78f5vzx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-w54s7z-sj4xc78f5vzx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-w54s7z/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:459]: failed to create nodepool nps-420-422 at version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc00093ba00>: 
    failed to create NodePool nps-420-422: failed starting nodepool creation "nps-420-422" for cluster "np-skip-minor-w54s7z" in resourcegroup="rg-np-skip-minor-w54s7z-sj4xc78f5vzx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-w54s7z-sj4xc78f5vzx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-w54s7z/nodePools/nps-420-422
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: InvalidRequestContent; detail message Invalid value: "<ocp-nightly>": must be at least 4.20.8; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20 · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-8gnl2p" for cluster "cluster-nightly-4-20-8gnl2p"
Unexpected error:
    <*fmt.wrapError | 0xc001386c00>: 
    failed to create NodePool np-8gnl2p: failed starting nodepool creation "np-8gnl2p" for cluster "cluster-nightly-4-20-8gnl2p" in resourcegroup="rg-nightly-4-20-pwndrzvtlxs6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-pwndrzvtlxs6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-8gnl2p/nodePools/np-8gnl2p
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:175]: failed to create node pool "np-8gnl2p" for cluster "cluster-nightly-4-20-8gnl2p"
Unexpected error:
    <*fmt.wrapError | 0xc001386c00>: 
    failed to create NodePool np-8gnl2p: failed starting nodepool creation "np-8gnl2p" for cluster "cluster-nightly-4-20-8gnl2p" in resourcegroup="rg-nightly-4-20-pwndrzvtlxs6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-pwndrzvtlxs6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-8gnl2p/nodePools/np-8gnl2p
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidRequestContent
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidRequestContent",
        "message": "Invalid value: \"4.20.0-0.nightly-multi-2026-09-16-210648\": must be at least 4.20.8",
        "target": "properties.version.id"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
2h 30m 22s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
brazilsouth n/a passed n/a 0 n/a
2h 27m 6s branch-ci-Azure-ARO-HCP-main-e2e-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
brazilsouth e2e failed n/a 1
e2e (1)
ERROR CODE: InternalServerError; detail code OperationNotAllowed; detail message Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-sgbzk-<id>' since another operation is in progress.; provider Microsoft.Compute
02:24:25 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:204]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00160b7d0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-rzlnqd88ksbr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0a92b991-1627-4a97-9fea-a9edfc1c8c70
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0a92b991-1627-4a97-9fea-a9edfc1c8c70",
      "name": "0a92b991-1627-4a97-9fea-a9edfc1c8c70",
      "status": "Failed",
      "startTime": "2026-09-19T02:54:29.386299179Z",
      "endTime": "2026-09-19T03:13:30.082275878Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    encountered errors downloading console logs:
      - VM "cilium-cluster-cilium-np-sgbzk-2tmm2": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-sgbzk-2tmm2" after 3 attempts, last error: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-rzlnqd88ksbr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-sgbzk-2tmm2/retrieveBootDiagnosticsData
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: OperationNotAllowed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "OperationNotAllowed",
        "message": "Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-sgbzk-2tmm2' since another operation is in progress."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:204]: failed to verify nodes are Ready with Cilium CNI for cluster "cilium-cluster"
Unexpected error:
    <*errors.joinError | 0xc00160b7d0>: 
    failed to create NodePool cilium-np: failed waiting for nodepool="cilium-np" for cluster "cilium-cluster" in resourcegroup="complex-cilium-kv-rzlnqd88ksbr" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0a92b991-1627-4a97-9fea-a9edfc1c8c70
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/brazilsouth/hcpOperationStatuses/0a92b991-1627-4a97-9fea-a9edfc1c8c70",
      "name": "0a92b991-1627-4a97-9fea-a9edfc1c8c70",
      "status": "Failed",
      "startTime": "2026-09-19T02:54:29.386299179Z",
      "endTime": "2026-09-19T03:13:30.082275878Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    encountered errors downloading console logs:
      - VM "cilium-cluster-cilium-np-sgbzk-2tmm2": failed to retrieve boot diagnostics data for VM "cilium-cluster-cilium-np-sgbzk-2tmm2" after 3 attempts, last error: POST https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-rzlnqd88ksbr--managed/providers/Microsoft.Compute/virtualMachines/cilium-cluster-cilium-np-sgbzk-2tmm2/retrieveBootDiagnosticsData
    --------------------------------------------------------------------------------
    RESPONSE 409: 409 Conflict
    ERROR CODE: OperationNotAllowed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "OperationNotAllowed",
        "message": "Operation 'retrieveBootDiagnosticsData' is not allowed on VM 'cilium-cluster-cilium-np-sgbzk-2tmm2' since another operation is in progress."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
2h 9m 12s periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel
post-goodmerged PR
No additional run metadata captured
uksouth e2e failed n/a 1
e2e (1)
failed to create nodepool <nodepool>: failed waiting for nodepool="<nodepool>" for cluster "<cluster>" in resourcegroup="<resource-group>" to finish creating: GET <url>
02:00:32 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite prod/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00381c7e0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-8j5t9t6fn8sk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d87907d8-e7ff-4012-ba96-a28fad371f71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d87907d8-e7ff-4012-ba96-a28fad371f71",
      "name": "d87907d8-e7ff-4012-ba96-a28fad371f71",
      "status": "Failed",
      "startTime": "2026-09-19T03:13:36.546108406Z",
      "endTime": "2026-09-19T03:32:43.229867603Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:127]: failed to create node pools
Expected success, but got an error:
    <*fmt.wrapError | 0xc00381c7e0>: 
    failed to create NodePool np-deletable: failed waiting for nodepool="np-deletable" for cluster "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-8j5t9t6fn8sk" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d87907d8-e7ff-4012-ba96-a28fad371f71
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOperationStatuses/d87907d8-e7ff-4012-ba96-a28fad371f71",
      "name": "d87907d8-e7ff-4012-ba96-a28fad371f71",
      "status": "Failed",
      "startTime": "2026-09-19T03:13:36.546108406Z",
      "endTime": "2026-09-19T03:32:43.229867603Z",
      "error": {
        "code": "InternalServerError",
        "message": "node pool creation did not complete before the deadline"
      }
    }
    --------------------------------------------------------------------------------
    
    ...

Environment: STG

Time (UTC)RuntimeJobRegionFailed atResultPRFailed testsDetails
34m 5s periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel-ocp-nightly
post-goodmerged PR
No additional run metadata captured
uksouth e2e failed n/a 52
e2e (52)
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000fa6100>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-jrsfr4h4jx7t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-jrsfr4h4jx7t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000fa6100>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-jrsfr4h4jx7t": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-jrsfr4h4jx7t/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc00063e160>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-mfd2k7xxpjd7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-mfd2k7xxpjd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc00063e160>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-mfd2k7xxpjd7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-mfd2k7xxpjd7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc0005b4c80>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-56knt66zd5sq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-56knt66zd5sq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc0005b4c80>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-56knt66zd5sq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-56knt66zd5sq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000f8e0e0>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-5q8j5rg67ft5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-5q8j5rg67ft5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc000f8e0e0>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-5q8j5rg67ft5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-5q8j5rg67ft5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc0011582c0>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-mnshgv6mq89m": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-mnshgv6mq89m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc0011582c0>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-mnshgv6mq89m": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-mnshgv6mq89m/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to delete an HCP cluster whose managed identities were deleted first · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000b8cb80>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-l2t9lwhx5kdj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-l2t9lwhx5kdj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000b8cb80>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-l2t9lwhx5kdj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-l2t9lwhx5kdj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000f06100>: 
    failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-sgzzfdtmcbtj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-sgzzfdtmcbtj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc000f06100>: 
    failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-sgzzfdtmcbtj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-sgzzfdtmcbtj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000d18240>: 
    failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2r2jrs6fg7s9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2r2jrs6fg7s9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc000d18240>: 
    failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-2r2jrs6fg7s9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-2r2jrs6fg7s9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.0 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-0-rhdtd9zkrfsz/cluster-nightly-5-0-jmdgbq should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c58aa0>: 
    failed to create HCP cluster cluster-nightly-5-0-jmdgbq: failed starting cluster creation "cluster-nightly-5-0-jmdgbq" in resourcegroup="rg-nightly-5-0-rhdtd9zkrfsz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-rhdtd9zkrfsz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-0-jmdgbq
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-0-rhdtd9zkrfsz/cluster-nightly-5-0-jmdgbq should provision
Unexpected error:
    <*fmt.wrapError | 0xc000c58aa0>: 
    failed to create HCP cluster cluster-nightly-5-0-jmdgbq: failed starting cluster creation "cluster-nightly-5-0-jmdgbq" in resourcegroup="rg-nightly-5-0-rhdtd9zkrfsz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-0-rhdtd9zkrfsz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-0-jmdgbq
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to cordon, drain, and uncordon a node in an HCP cluster · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e583e0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-2jb5fhv675q2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-2jb5fhv675q2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e583e0>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-2jb5fhv675q2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-2jb5fhv675q2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc0009a4540>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-qqzhhqdh5xkq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-qqzhhqdh5xkq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc0009a4540>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-qqzhhqdh5xkq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-qqzhhqdh5xkq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:105]: failed to begin creating HCP cluster "admin-cred-lifecycle-df2r22"
Unexpected error:
    <*fmt.wrapError | 0xc00107e160>: 
    failed starting cluster creation "admin-cred-lifecycle-df2r22" in resourcegroup="admin-credential-lifecycle-test-d8n845mt5q44": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-d8n845mt5q44/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-df2r22
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:105]: failed to begin creating HCP cluster "admin-cred-lifecycle-df2r22"
Unexpected error:
    <*fmt.wrapError | 0xc00107e160>: 
    failed starting cluster creation "admin-cred-lifecycle-df2r22" in resourcegroup="admin-credential-lifecycle-test-d8n845mt5q44": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-d8n845mt5q44/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-df2r22
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc001096160>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-bjjrbq6n7zxf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-bjjrbq6n7zxf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc001096160>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-bjjrbq6n7zxf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-bjjrbq6n7zxf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 5.0 minor to 5.1 minor · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-7xq4bv" at install version 5.0.0-0.nightly-multi-2026-09-16-151028
Unexpected error:
    <*fmt.wrapError | 0xc000f243a0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-7xq4bv: failed starting cluster creation "cp-ystream-upgrade-5-1-7xq4bv" in resourcegroup="rg-cp-ystream-upgrade-5-1-hp5qbbg84rbl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-hp5qbbg84rbl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-7xq4bv
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-1-7xq4bv" at install version 5.0.0-0.nightly-multi-2026-09-16-151028
Unexpected error:
    <*fmt.wrapError | 0xc000f243a0>: 
    failed to create HCP cluster cp-ystream-upgrade-5-1-7xq4bv: failed starting cluster creation "cp-ystream-upgrade-5-1-7xq4bv" in resourcegroup="rg-cp-ystream-upgrade-5-1-hp5qbbg84rbl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-1-hp5qbbg84rbl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-1-7xq4bv
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000d501a0>: 
    failed to create HCP cluster ea-list6ggcpm: failed starting cluster creation "ea-list6ggcpm" in resourcegroup="ea-list-52ppn4fhvbz4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-52ppn4fhvbz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-list6ggcpm
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc000d501a0>: 
    failed to create HCP cluster ea-list6ggcpm: failed starting cluster creation "ea-list6ggcpm" in resourcegroup="ea-list-52ppn4fhvbz4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-52ppn4fhvbz4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-list6ggcpm
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 4.23 minor · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-mbqqb8" at install version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc0009a0160>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-mbqqb8: failed starting cluster creation "cp-ystream-upgrade-4-23-mbqqb8" in resourcegroup="rg-cp-ystream-upgrade-4-23-ndnktb84bpss": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-ndnktb84bpss/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-mbqqb8
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-23-mbqqb8" at install version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc0009a0160>: 
    failed to create HCP cluster cp-ystream-upgrade-4-23-mbqqb8: failed starting cluster creation "cp-ystream-upgrade-4-23-mbqqb8" in resourcegroup="rg-cp-ystream-upgrade-4-23-ndnktb84bpss": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-23-ndnktb84bpss/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-23-mbqqb8
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001630120>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-lb88jl5tzm57": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-lb88jl5tzm57/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc001630120>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-lb88jl5tzm57": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-lb88jl5tzm57/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ba8100>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-n57ssgdqmbg7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-n57ssgdqmbg7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000ba8100>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-n57ssgdqmbg7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-n57ssgdqmbg7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc0010061a0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-f2pr5885x44k": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-f2pr5885x44k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc0010061a0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-f2pr5885x44k": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-f2pr5885x44k/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000b94360>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-8999wm6hwv4j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8999wm6hwv4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000b94360>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-8999wm6hwv4j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-8999wm6hwv4j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000910d60>: 
    failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-qnslkbxblpk7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-qnslkbxblpk7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000910d60>: 
    failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-qnslkbxblpk7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-qnslkbxblpk7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c6c480>: 
    failed to create HCP cluster np-dg-minor-829r99: failed starting cluster creation "np-dg-minor-829r99" in resourcegroup="rg-np-dg-minor-829r99-lcmrdp2wthfh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-829r99-lcmrdp2wthfh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-829r99
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c6c480>: 
    failed to create HCP cluster np-dg-minor-829r99: failed starting cluster creation "np-dg-minor-829r99" in resourcegroup="rg-np-dg-minor-829r99-lcmrdp2wthfh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-829r99-lcmrdp2wthfh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-829r99
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to list available HCP OpenShift versions and validate response content · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc0015347b0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc0015347b0>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 5.1 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-1-vc5fnpg6krpv/cluster-nightly-5-1-tgrjnq should provision
Unexpected error:
    <*fmt.wrapError | 0xc00086e2a0>: 
    failed to create HCP cluster cluster-nightly-5-1-tgrjnq: failed starting cluster creation "cluster-nightly-5-1-tgrjnq" in resourcegroup="rg-nightly-5-1-vc5fnpg6krpv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-vc5fnpg6krpv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-1-tgrjnq
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-5-1-vc5fnpg6krpv/cluster-nightly-5-1-tgrjnq should provision
Unexpected error:
    <*fmt.wrapError | 0xc00086e2a0>: 
    failed to create HCP cluster cluster-nightly-5-1-tgrjnq: failed starting cluster creation "cluster-nightly-5-1-tgrjnq" in resourcegroup="rg-nightly-5-1-vc5fnpg6krpv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-5-1-vc5fnpg6krpv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-5-1-tgrjnq
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000356220>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zn9jmj6hhhs8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zn9jmj6hhhs8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000356220>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-zn9jmj6hhhs8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-zn9jmj6hhhs8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.23 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-23-gq5d8dl5q8cm/cluster-nightly-4-23-mb5khv should provision
Unexpected error:
    <*fmt.wrapError | 0xc0009a0320>: 
    failed to create HCP cluster cluster-nightly-4-23-mb5khv: failed starting cluster creation "cluster-nightly-4-23-mb5khv" in resourcegroup="rg-nightly-4-23-gq5d8dl5q8cm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-gq5d8dl5q8cm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-23-mb5khv
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-23-gq5d8dl5q8cm/cluster-nightly-4-23-mb5khv should provision
Unexpected error:
    <*fmt.wrapError | 0xc0009a0320>: 
    failed to create HCP cluster cluster-nightly-4-23-mb5khv: failed starting cluster creation "cluster-nightly-4-23-mb5khv" in resourcegroup="rg-nightly-4-23-gq5d8dl5q8cm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-23-gq5d8dl5q8cm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-23-mb5khv
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc00096c720>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-xbsnfj694jv9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-xbsnfj694jv9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc00096c720>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-xbsnfj694jv9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-xbsnfj694jv9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0009c0260>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-jsqrlcdx92hv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-jsqrlcdx92hv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc0009c0260>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-jsqrlcdx92hv": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-jsqrlcdx92hv/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc0010d40c0>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-8lzz4fz9ld22": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-8lzz4fz9ld22/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc0010d40c0>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-8lzz4fz9ld22": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-8lzz4fz9ld22/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc0010263e0>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-q9t7xgddb42v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-q9t7xgddb42v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc0010263e0>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-q9t7xgddb42v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-q9t7xgddb42v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001136240>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-66v48h2xktkt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-66v48h2xktkt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc001136240>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-66v48h2xktkt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-66v48h2xktkt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-9hdq9v with version 4.21.0-0.nightly-multi-2026-09-17-004700
Unexpected error:
    <*fmt.wrapError | 0xc00011b300>: 
    failed to create HCP cluster np-version-upgrade-cluster-9hdq9v: failed starting cluster creation "np-version-upgrade-cluster-9hdq9v" in resourcegroup="rg-np-version-upgrade-9hdq9v-pxwzxjdnh87j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9hdq9v-pxwzxjdnh87j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-9hdq9v
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-9hdq9v with version 4.21.0-0.nightly-multi-2026-09-17-004700
Unexpected error:
    <*fmt.wrapError | 0xc00011b300>: 
    failed to create HCP cluster np-version-upgrade-cluster-9hdq9v: failed starting cluster creation "np-version-upgrade-cluster-9hdq9v" in resourcegroup="rg-np-version-upgrade-9hdq9v-pxwzxjdnh87j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-9hdq9v-pxwzxjdnh87j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-9hdq9v
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-fgqjd9" at install version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc000ae4800>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-fgqjd9: failed starting cluster creation "cp-ystream-upgrade-4-21-fgqjd9" in resourcegroup="rg-cp-ystream-upgrade-4-21-gg4rdn5pv5xg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-gg4rdn5pv5xg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-fgqjd9
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-fgqjd9" at install version 4.20.0-0.nightly-multi-2026-09-16-210648
Unexpected error:
    <*fmt.wrapError | 0xc000ae4800>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-fgqjd9: failed starting cluster creation "cp-ystream-upgrade-4-21-fgqjd9" in resourcegroup="rg-cp-ystream-upgrade-4-21-gg4rdn5pv5xg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-gg4rdn5pv5xg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-fgqjd9
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0010bc3a0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-8wqm69gkd22v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-8wqm69gkd22v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc0010bc3a0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-8wqm69gkd22v": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-8wqm69gkd22v/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.22 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-22-rxbnc27m79nk/cluster-nightly-4-22-pvbckf should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010102c0>: 
    failed to create HCP cluster cluster-nightly-4-22-pvbckf: failed starting cluster creation "cluster-nightly-4-22-pvbckf" in resourcegroup="rg-nightly-4-22-rxbnc27m79nk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-rxbnc27m79nk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-22-pvbckf
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-22-rxbnc27m79nk/cluster-nightly-4-22-pvbckf should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010102c0>: 
    failed to create HCP cluster cluster-nightly-4-22-pvbckf: failed starting cluster creation "cluster-nightly-4-22-pvbckf" in resourcegroup="rg-nightly-4-22-rxbnc27m79nk": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-22-rxbnc27m79nk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-22-pvbckf
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000f120c0>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-bff9c2jt9jgp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-bff9c2jt9jgp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000f120c0>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-bff9c2jt9jgp": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-bff9c2jt9jgp/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e7c400>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-r7vfdmhdqlkx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-r7vfdmhdqlkx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000e7c400>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-r7vfdmhdqlkx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-r7vfdmhdqlkx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc00036a5c0>: 
    failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-9g9z8hvhct9l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-9g9z8hvhct9l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc00036a5c0>: 
    failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-9g9z8hvhct9l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-9g9z8hvhct9l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000ac24e0>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-977lq5c64l4r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-977lq5c64l4r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc000ac24e0>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-977lq5c64l4r": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-977lq5c64l4r/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000922f20>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-dmkwprhgsz8l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-dmkwprhgsz8l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000922f20>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-dmkwprhgsz8l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-dmkwprhgsz8l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-t2t5bm at version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc001694340>: 
    failed to create HCP cluster np-skip-minor-t2t5bm: failed starting cluster creation "np-skip-minor-t2t5bm" in resourcegroup="rg-np-skip-minor-t2t5bm-svk68nntb69q": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-t2t5bm-svk68nntb69q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-t2t5bm
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-t2t5bm at version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc001694340>: 
    failed to create HCP cluster np-skip-minor-t2t5bm: failed starting cluster creation "np-skip-minor-t2t5bm" in resourcegroup="rg-np-skip-minor-t2t5bm-svk68nntb69q": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-t2t5bm-svk68nntb69q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-t2t5bm
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.21 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-21-8ssgwkgs2vpn/cluster-nightly-4-21-ghtt6c should provision
Unexpected error:
    <*fmt.wrapError | 0xc0014da240>: 
    failed to create HCP cluster cluster-nightly-4-21-ghtt6c: failed starting cluster creation "cluster-nightly-4-21-ghtt6c" in resourcegroup="rg-nightly-4-21-8ssgwkgs2vpn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-8ssgwkgs2vpn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-21-ghtt6c
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-21-8ssgwkgs2vpn/cluster-nightly-4-21-ghtt6c should provision
Unexpected error:
    <*fmt.wrapError | 0xc0014da240>: 
    failed to create HCP cluster cluster-nightly-4-21-ghtt6c: failed starting cluster creation "cluster-nightly-4-21-ghtt6c" in resourcegroup="rg-nightly-4-21-8ssgwkgs2vpn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-21-8ssgwkgs2vpn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-21-ghtt6c
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc0011ae2e0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-nvqdsqj6pgcq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-nvqdsqj6pgcq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc0011ae2e0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-nvqdsqj6pgcq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-nvqdsqj6pgcq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc0010bd380>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-trjdkffls2tk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc0010bd380>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-trjdkffls2tk/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000c04620>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-bcb7tk5bk7cd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-bcb7tk5bk7cd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc000c04620>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-bcb7tk5bk7cd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-bcb7tk5bk7cd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP nightly channel for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-20-cscfjhjdwf5g/cluster-nightly-4-20-p8t827 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001588d60>: 
    failed to create HCP cluster cluster-nightly-4-20-p8t827: failed starting cluster creation "cluster-nightly-4-20-p8t827" in resourcegroup="rg-nightly-4-20-cscfjhjdwf5g": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-cscfjhjdwf5g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-p8t827
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-nightly-4-20-cscfjhjdwf5g/cluster-nightly-4-20-p8t827 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001588d60>: 
    failed to create HCP cluster cluster-nightly-4-20-p8t827: failed starting cluster creation "cluster-nightly-4-20-p8t827" in resourcegroup="rg-nightly-4-20-cscfjhjdwf5g": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-nightly-4-20-cscfjhjdwf5g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-nightly-4-20-p8t827
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000c92140>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-mw2wrjlf7jq6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-mw2wrjlf7jq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000c92140>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-mw2wrjlf7jq6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-mw2wrjlf7jq6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-gll9jp" at install version 4.21.0-0.nightly-multi-2026-09-17-004700
Unexpected error:
    <*fmt.wrapError | 0xc000c1e660>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-gll9jp: failed starting cluster creation "cp-ystream-upgrade-4-22-gll9jp" in resourcegroup="rg-cp-ystream-upgrade-4-22-b98g5ck8279p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-b98g5ck8279p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-gll9jp
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-gll9jp" at install version 4.21.0-0.nightly-multi-2026-09-17-004700
Unexpected error:
    <*fmt.wrapError | 0xc000c1e660>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-gll9jp: failed starting cluster creation "cp-ystream-upgrade-4-22-gll9jp" in resourcegroup="rg-cp-ystream-upgrade-4-22-b98g5ck8279p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-b98g5ck8279p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-gll9jp
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-mmw4ln" at install version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc0003d5800>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-mmw4ln: failed starting cluster creation "cp-ystream-upgrade-5-0-mmw4ln" in resourcegroup="rg-cp-ystream-upgrade-5-0-5b5ddg7grm6j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-5b5ddg7grm6j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-mmw4ln
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-mmw4ln" at install version 4.22.0-0.nightly-multi-2026-09-17-071442
Unexpected error:
    <*fmt.wrapError | 0xc0003d5800>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-mmw4ln: failed starting cluster creation "cp-ystream-upgrade-5-0-mmw4ln" in resourcegroup="rg-cp-ystream-upgrade-5-0-5b5ddg7grm6j": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-5b5ddg7grm6j/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-mmw4ln
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000f90420>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-bm5ss97nkdlm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-bm5ss97nkdlm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000f90420>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-bm5ss97nkdlm": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-bm5ss97nkdlm/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc0010ac0e0>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-s8jj6wkd7zgx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-s8jj6wkd7zgx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc0010ac0e0>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-s8jj6wkd7zgx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-s8jj6wkd7zgx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
23:00:33 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00068d740>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-2thnp8vdw5vt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-2thnp8vdw5vt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc00068d740>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-2thnp8vdw5vt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-2thnp8vdw5vt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
15m 43s periodic-ci-Azure-ARO-HCP-main-periodic-stage-e2e-parallel
post-goodmerged PR
No additional run metadata captured
uksouth e2e failed n/a 56
e2e (56)
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.21.z to 4.21.zLatest · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-st8fgn with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000a882e0>: 
    failed to create HCP cluster np-version-upgrade-cluster-st8fgn: failed starting cluster creation "np-version-upgrade-cluster-st8fgn" in resourcegroup="rg-np-version-upgrade-st8fgn-rzpd48n7mdnl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-st8fgn-rzpd48n7mdnl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-st8fgn
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-st8fgn with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000a882e0>: 
    failed to create HCP cluster np-version-upgrade-cluster-st8fgn: failed starting cluster creation "np-version-upgrade-cluster-st8fgn" in resourcegroup="rg-np-version-upgrade-st8fgn-rzpd48n7mdnl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-st8fgn-rzpd48n7mdnl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-st8fgn
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.22 minor to 5.0 minor · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-rl4rg9" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc0017e8920>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-rl4rg9: failed starting cluster creation "cp-ystream-upgrade-5-0-rl4rg9" in resourcegroup="rg-cp-ystream-upgrade-5-0-bplmhxcdrbl5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-bplmhxcdrbl5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-rl4rg9
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-5-0-rl4rg9" at install version 4.22
Unexpected error:
    <*fmt.wrapError | 0xc0017e8920>: 
    failed to create HCP cluster cp-ystream-upgrade-5-0-rl4rg9: failed starting cluster creation "cp-ystream-upgrade-5-0-rl4rg9" in resourcegroup="rg-cp-ystream-upgrade-5-0-bplmhxcdrbl5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-5-0-bplmhxcdrbl5/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-5-0-rl4rg9
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc0010589a0>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-gfnjbcrxm92q": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-gfnjbcrxm92q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:111]: failed to create HCP cluster "cidr-connectivity-test" with authorized CIDRs
Unexpected error:
    <*fmt.wrapError | 0xc0010589a0>: 
    failed to create HCP cluster cidr-connectivity-test: failed starting cluster creation "cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-gfnjbcrxm92q": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/e2e-cidr-connectivity-gfnjbcrxm92q/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cidr-connectivity-test
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.21 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-4mtmtq" with version 4.21.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000abe620>: 
    failed to create HCP cluster cluster-zstream-4-21-4mtmtq: failed starting cluster creation "cluster-zstream-4-21-4mtmtq" in resourcegroup="rg-zstream-upgrade-4-21-l8n58x42msl8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-l8n58x42msl8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-4mtmtq
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-21-4mtmtq" with version 4.21.33 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000abe620>: 
    failed to create HCP cluster cluster-zstream-4-21-4mtmtq: failed starting cluster creation "cluster-zstream-4-21-4mtmtq" in resourcegroup="rg-zstream-upgrade-4-21-l8n58x42msl8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-21-l8n58x42msl8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-21-4mtmtq
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled up to replica limits · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0013ec2a0>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-pvgssfd4lms2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-pvgssfd4lms2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:102]: failed to create HCP cluster np-autoscale-cluster
Unexpected error:
    <*fmt.wrapError | 0xc0013ec2a0>: 
    failed to create HCP cluster np-autoscale-cluster: failed starting cluster creation "np-autoscale-cluster" in resourcegroup="np-autoscaling-pvgssfd4lms2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/np-autoscaling-pvgssfd4lms2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-autoscale-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to delete a nodepool · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b14180>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-psjfv4lsrrfw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-psjfv4lsrrfw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_delete.go:80]: failed to create HCP cluster np-delete-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000b14180>: 
    failed to create HCP cluster np-delete-hcp-cluster: failed starting cluster creation "np-delete-hcp-cluster" in resourcegroup="nodepool-delete-psjfv4lsrrfw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/nodepool-delete-psjfv4lsrrfw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-delete-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 5.0 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-mmd4pgsbmbps/cluster-candidate-5-0-4xcnlw should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fd03a0>: 
    failed to create HCP cluster cluster-candidate-5-0-4xcnlw: failed starting cluster creation "cluster-candidate-5-0-4xcnlw" in resourcegroup="rg-candidate-5-0-mmd4pgsbmbps": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mmd4pgsbmbps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-4xcnlw
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-5-0-mmd4pgsbmbps/cluster-candidate-5-0-4xcnlw should provision
Unexpected error:
    <*fmt.wrapError | 0xc000fd03a0>: 
    failed to create HCP cluster cluster-candidate-5-0-4xcnlw: failed starting cluster creation "cluster-candidate-5-0-4xcnlw" in resourcegroup="rg-candidate-5-0-mmd4pgsbmbps": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-5-0-mmd4pgsbmbps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-5-0-4xcnlw
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should upgrade a nodepool to a version without Cincinnati upgrade edge z-stream upgrade without Cincinnati edge in 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-rlcklb
Unexpected error:
    <*fmt.wrapError | 0xc00101e320>: 
    failed to create HCP cluster np-noedge-rlcklb: failed starting cluster creation "np-noedge-rlcklb" in resourcegroup="rg-np-noedge-rlcklb-hd6g5qz9bvfl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-rlcklb-hd6g5qz9bvfl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-rlcklb
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:301]: failed to create HCP cluster np-noedge-rlcklb
Unexpected error:
    <*fmt.wrapError | 0xc00101e320>: 
    failed to create HCP cluster np-noedge-rlcklb: failed starting cluster creation "np-noedge-rlcklb" in resourcegroup="rg-np-noedge-rlcklb-hd6g5qz9bvfl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-noedge-rlcklb-hd6g5qz9bvfl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-noedge-rlcklb
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc0009e02a0>: 
    failed to create HCP cluster ea-list7bqcqx: failed starting cluster creation "ea-list7bqcqx" in resourcegroup="ea-list-t8h98fdlrftn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-t8h98fdlrftn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-list7bqcqx
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:87]: failed to create HCP cluster for external auth list test
Unexpected error:
    <*fmt.wrapError | 0xc0009e02a0>: 
    failed to create HCP cluster ea-list7bqcqx: failed starting cluster creation "ea-list7bqcqx" in resourcegroup="ea-list-t8h98fdlrftn": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ea-list-t8h98fdlrftn/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-list7bqcqx
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.22 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-zdkhd697924f/cluster-candidate-4-22-pbsdf9 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010603c0>: 
    failed to create HCP cluster cluster-candidate-4-22-pbsdf9: failed starting cluster creation "cluster-candidate-4-22-pbsdf9" in resourcegroup="rg-candidate-4-22-zdkhd697924f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-zdkhd697924f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-22-pbsdf9
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-22-zdkhd697924f/cluster-candidate-4-22-pbsdf9 should provision
Unexpected error:
    <*fmt.wrapError | 0xc0010603c0>: 
    failed to create HCP cluster cluster-candidate-4-22-pbsdf9: failed starting cluster creation "cluster-candidate-4-22-pbsdf9" in resourcegroup="rg-candidate-4-22-zdkhd697924f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-22-zdkhd697924f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-22-pbsdf9
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to create an HCP cluster then delete it by deleting the customer resource group · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f1e280>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-8npg9bvv7cwh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-8npg9bvv7cwh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_cx_rg.go:90]: failed to create HCP cluster "cx-rg-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000f1e280>: 
    failed to create HCP cluster cx-rg-hcp-cluster: failed starting cluster creation "cx-rg-hcp-cluster" in resourcegroup="cx-rg-cluster-8npg9bvv7cwh": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cx-rg-cluster-8npg9bvv7cwh/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cx-rg-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test ARO HCP Service should successfully provision cluster when MI role assignments are added after cluster creation · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc001584810>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-jcl645m442x6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delayed_role_assignments.go:140]: failed to start cluster creation
Unexpected error:
    <*exported.ResponseError | 0xc001584810>: 
    PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delayed-rbac-jcl645m442x6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/delayed-rbac-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000c80280>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-dm69g797vr92": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-dm69g797vr92/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:98]: failed to create HCP cluster "private-kv-cluster" with private keyvault
Unexpected error:
    <*fmt.wrapError | 0xc000c80280>: 
    failed starting cluster creation "private-kv-cluster" in resourcegroup="private-keyvault-dm69g797vr92": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-keyvault-dm69g797vr92/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kv-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.20.zLatest · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-z2bllz with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000d9c140>: 
    failed to create HCP cluster np-version-upgrade-cluster-z2bllz: failed starting cluster creation "np-version-upgrade-cluster-z2bllz" in resourcegroup="rg-np-version-upgrade-z2bllz-r8mpwp6kjwld": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z2bllz-r8mpwp6kjwld/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-z2bllz
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-z2bllz with version 4.20.38
Unexpected error:
    <*fmt.wrapError | 0xc000d9c140>: 
    failed to create HCP cluster np-version-upgrade-cluster-z2bllz: failed starting cluster creation "np-version-upgrade-cluster-z2bllz" in resourcegroup="rg-np-version-upgrade-z2bllz-r8mpwp6kjwld": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-z2bllz-r8mpwp6kjwld/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-z2bllz
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc001208520>: 
    failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-plmnmvlvbjz8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-plmnmvlvbjz8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:291]: failed to create HCP cluster node-limit-cluster with MaxNodesTotal limit
Unexpected error:
    <*fmt.wrapError | 0xc001208520>: 
    failed to create HCP cluster node-limit-cluster: failed starting cluster creation "node-limit-cluster" in resourcegroup="node-limit-test-plmnmvlvbjz8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/node-limit-test-plmnmvlvbjz8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/node-limit-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to create a no-CNI private cluster with a private key vault, a nodepool and install cilium CNI successfully · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000fac240>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-qxfdp7vf452w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-qxfdp7vf452w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_complex_cilium_kv.go:106]: failed to create HCP cluster "cilium-cluster" with no CNI and private etcd
Unexpected error:
    <*fmt.wrapError | 0xc000fac240>: 
    failed starting cluster creation "cilium-cluster" in resourcegroup="complex-cilium-kv-qxfdp7vf452w": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/complex-cilium-kv-qxfdp7vf452w/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc0012a4140>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-n97st4z98mw9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-n97st4z98mw9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: failed to create HCP cluster for patch-name test
Unexpected error:
    <*fmt.wrapError | 0xc0012a4140>: 
    failed to create HCP cluster patch-name-cluster: failed starting cluster creation "patch-name-cluster" in resourcegroup="patch-name-n97st4z98mw9": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-name-n97st4z98mw9/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-name-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000b38760>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-vjhljcrb66jz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vjhljcrb66jz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:113]: failed to create HCP cluster with ImageDigestMirrors
Unexpected error:
    <*fmt.wrapError | 0xc000b38760>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed starting cluster creation "idms-e2e-hcp-cluster" in resourcegroup="idms-vjhljcrb66jz": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/idms-vjhljcrb66jz/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/idms-e2e-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.22 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-8x697n" with version 4.22.13 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0011fe400>: 
    failed to create HCP cluster cluster-zstream-4-22-8x697n: failed starting cluster creation "cluster-zstream-4-22-8x697n" in resourcegroup="rg-zstream-upgrade-4-22-pk2l8wgvm8xq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-pk2l8wgvm8xq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-8x697n
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-22-8x697n" with version 4.22.13 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc0011fe400>: 
    failed to create HCP cluster cluster-zstream-4-22-8x697n: failed starting cluster creation "cluster-zstream-4-22-8x697n" in resourcegroup="rg-zstream-upgrade-4-22-pk2l8wgvm8xq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-22-pk2l8wgvm8xq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-22-8x697n
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc00180a140>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-lf8mh5l2h4w4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-lf8mh5l2h4w4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:96]: failed to create HCP cluster for TLS endpoint test
Unexpected error:
    <*fmt.wrapError | 0xc00180a140>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed starting cluster creation "tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-lf8mh5l2h4w4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/tls-endpoint-cluster-lf8mh5l2h4w4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/tls-endpoint-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000eca240>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-gs6t8h7vjq6x": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-gs6t8h7vjq6x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:111]: failed to begin creation of first cluster "basic-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc000eca240>: 
    failed starting cluster creation "basic-hcp-cluster" in resourcegroup="customer-rg-gs6t8h7vjq6x": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/customer-rg-gs6t8h7vjq6x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should create a cluster with private ingress using v20260901preview and verify the ingress is internal · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000e901c0>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-9jdv7b4h7nst": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-9jdv7b4h7nst/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_ingress.go:87]: failed to create HCP cluster "private-ingress" with private ingress
Unexpected error:
    <*fmt.wrapError | 0xc000e901c0>: 
    failed to create HCP cluster private-ingress: failed starting cluster creation "private-ingress" in resourcegroup="private-ingress-9jdv7b4h7nst": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-ingress-9jdv7b4h7nst/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-ingress
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc0009b62c0>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-zf8dqph82cps": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-zf8dqph82cps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:152]: failed to create HCP cluster version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc0009b62c0>: 
    failed starting cluster creation "cluster-ver-4-20" in resourcegroup="rg-cluster-back-version-zf8dqph82cps": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-zf8dqph82cps/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-ver-4-20
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should upgrade and update a nodepool from 4.20.z to 4.21.zLatest · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-njr6vs with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000a66180>: 
    failed to create HCP cluster np-version-upgrade-cluster-njr6vs: failed starting cluster creation "np-version-upgrade-cluster-njr6vs" in resourcegroup="rg-np-version-upgrade-njr6vs-szft45d6n4n7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-njr6vs-szft45d6n4n7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-njr6vs
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:146]: failed to create HCP cluster np-version-upgrade-cluster-njr6vs with version 4.21.34
Unexpected error:
    <*fmt.wrapError | 0xc000a66180>: 
    failed to create HCP cluster np-version-upgrade-cluster-njr6vs: failed starting cluster creation "np-version-upgrade-cluster-njr6vs" in resourcegroup="rg-np-version-upgrade-njr6vs-szft45d6n4n7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-version-upgrade-njr6vs-szft45d6n4n7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-version-upgrade-cluster-njr6vs
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000864160>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-hdjs9ld55ftw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-hdjs9ld55ftw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:161]: failed to create HCP cluster for patch-tags test
Unexpected error:
    <*fmt.wrapError | 0xc000864160>: 
    failed to create HCP cluster patch-tags-cluster: failed starting cluster creation "patch-tags-cluster" in resourcegroup="patch-tags-hdjs9ld55ftw": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/patch-tags-hdjs9ld55ftw/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/patch-tags-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc0011182a0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-nchtwmv686zd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-nchtwmv686zd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: failed to create HCP cluster with image registry disabled
Unexpected error:
    <*fmt.wrapError | 0xc0011182a0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed starting cluster creation "disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-nchtwmv686zd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/disabled-image-registry-nchtwmv686zd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/disabled-image-registry-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to create a cluster and node pools with aggregated advanced features · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000f080c0>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-w5275tjnvr7x": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-w5275tjnvr7x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_feature_aggregation.go:207]: failed to create HCP cluster "agg-cluster" with aggregated settings
Unexpected error:
    <*fmt.wrapError | 0xc000f080c0>: 
    failed starting cluster creation "agg-cluster" in resourcegroup="feature-aggregation-w5275tjnvr7x": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/feature-aggregation-w5275tjnvr7x/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/agg-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling and update the autoscaling configuration · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000736120>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-qsxzf9x22q4p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-qsxzf9x22q4p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:108]: failed to create HCP cluster "autoscaling-hcp-cluster" with custom autoscaling
Unexpected error:
    <*fmt.wrapError | 0xc000736120>: 
    failed to create HCP cluster autoscaling-hcp-cluster: failed starting cluster creation "autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-qsxzf9x22q4p": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/autoscaling-cluster-qsxzf9x22q4p/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/autoscaling-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-47cxqhrdfk9l/cluster-candidate-4-20-6gsr7t should provision
Unexpected error:
    <*fmt.wrapError | 0xc0006fc2c0>: 
    failed to create HCP cluster cluster-candidate-4-20-6gsr7t: failed starting cluster creation "cluster-candidate-4-20-6gsr7t" in resourcegroup="rg-candidate-4-20-47cxqhrdfk9l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-47cxqhrdfk9l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-20-6gsr7t
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-20-47cxqhrdfk9l/cluster-candidate-4-20-6gsr7t should provision
Unexpected error:
    <*fmt.wrapError | 0xc0006fc2c0>: 
    failed to create HCP cluster cluster-candidate-4-20-6gsr7t: failed starting cluster creation "cluster-candidate-4-20-6gsr7t" in resourcegroup="rg-candidate-4-20-47cxqhrdfk9l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-20-47cxqhrdfk9l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-20-6gsr7t
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc0012962a0>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-ps5nk968zfg7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-ps5nk968zfg7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:84]: failed to create HCP cluster ephemeral-disk
Unexpected error:
    <*fmt.wrapError | 0xc0012962a0>: 
    failed to create HCP cluster ephemeral-disk: failed starting cluster creation "ephemeral-disk" in resourcegroup="ephemeral-osdisk-ps5nk968zfg7": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ephemeral-osdisk-ps5nk968zfg7/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ephemeral-disk
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to create a HCP cluster and use cilium CNI plugin · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc00011bd80>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-h8vn4h6lkcgb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-h8vn4h6lkcgb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_cni_cilium.go:88]: failed to create HCP cluster "cilium-cl" without CNI
Unexpected error:
    <*fmt.wrapError | 0xc00011bd80>: 
    failed to create HCP cluster cilium-cl: failed starting cluster creation "cilium-cl" in resourcegroup="cni-cilium-h8vn4h6lkcgb": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/cni-cilium-h8vn4h6lkcgb/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cilium-cl
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:105]: failed to begin creating HCP cluster "admin-cred-lifecycle-sdjg24"
Unexpected error:
    <*fmt.wrapError | 0xc00090a560>: 
    failed starting cluster creation "admin-cred-lifecycle-sdjg24" in resourcegroup="admin-credential-lifecycle-test-vsjrtcg2tzwf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-vsjrtcg2tzwf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-sdjg24
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:105]: failed to begin creating HCP cluster "admin-cred-lifecycle-sdjg24"
Unexpected error:
    <*fmt.wrapError | 0xc00090a560>: 
    failed starting cluster creation "admin-cred-lifecycle-sdjg24" in resourcegroup="admin-credential-lifecycle-test-vsjrtcg2tzwf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/admin-credential-lifecycle-test-vsjrtcg2tzwf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/admin-cred-lifecycle-sdjg24
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.21 minor to 4.22 minor · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-mhbcss" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc0017da2a0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-mhbcss: failed starting cluster creation "cp-ystream-upgrade-4-22-mhbcss" in resourcegroup="rg-cp-ystream-upgrade-4-22-jw5p8wwrpcvq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-jw5p8wwrpcvq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-mhbcss
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-22-mhbcss" at install version 4.21
Unexpected error:
    <*fmt.wrapError | 0xc0017da2a0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-22-mhbcss: failed starting cluster creation "cp-ystream-upgrade-4-22-mhbcss" in resourcegroup="rg-cp-ystream-upgrade-4-22-jw5p8wwrpcvq": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-22-jw5p8wwrpcvq/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-22-mhbcss
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc00150a140>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-5ptp8l4jpkwf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-5ptp8l4jpkwf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: failed to create HCP cluster "hcp-cluster-np-128"
Unexpected error:
    <*fmt.wrapError | 0xc00150a140>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-5ptp8l4jpkwf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-5ptp8l4jpkwf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc00110c120>: 
    failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-snctqdvrlhkd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-snctqdvrlhkd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:92]: failed to create HCP cluster cluster-np-labels-taints
Unexpected error:
    <*fmt.wrapError | 0xc00110c120>: 
    failed to create HCP cluster cluster-np-labels-taints: failed starting cluster creation "cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-snctqdvrlhkd": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-labels-taints-snctqdvrlhkd/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-np-labels-taints
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test FIPS Mode Support with v2026-09-01 API should create an HCP cluster with FIPS mode enabled via cryptoRestrictions property · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000ea4120>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-vfc2ghqjt5wf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-vfc2ghqjt5wf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_fips_mode.go:87]: failed to create HCP cluster "fips-enabled-cluster" with cryptoRestrictions set to FIPS
Unexpected error:
    <*fmt.wrapError | 0xc000ea4120>: 
    failed starting cluster creation "fips-enabled-cluster" in resourcegroup="fips-enabled-vfc2ghqjt5wf": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/fips-enabled-vfc2ghqjt5wf/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/fips-enabled-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f9e120>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-gf5qzh2zbwbl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-gf5qzh2zbwbl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:107]: failed to create seed HCP cluster basic-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000f9e120>: 
    failed to create HCP cluster basic-cluster: failed starting cluster creation "basic-cluster" in resourcegroup="rg-cluster-nsg-subnet-reuse-gf5qzh2zbwbl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-nsg-subnet-reuse-gf5qzh2zbwbl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/basic-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should downgrade a nodepool version z-stream downgrade from 4.21.zLatest to 4.21.zPrevious · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000cb8180>: 
    failed to create HCP cluster np-downgrade-jjtxlv: failed starting cluster creation "np-downgrade-jjtxlv" in resourcegroup="rg-np-downgrade-jjtxlv-kz4bnmg99nnx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-jjtxlv-kz4bnmg99nnx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-jjtxlv
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:580]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000cb8180>: 
    failed to create HCP cluster np-downgrade-jjtxlv: failed starting cluster creation "np-downgrade-jjtxlv" in resourcegroup="rg-np-downgrade-jjtxlv-kz4bnmg99nnx": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-downgrade-jjtxlv-kz4bnmg99nnx/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-downgrade-jjtxlv
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000992220>: 
    failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-p8xqhgpdsfcg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-p8xqhgpdsfcg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:94]: failed to create HCP cluster for external auth test
Unexpected error:
    <*fmt.wrapError | 0xc000992220>: 
    failed to create HCP cluster ea-cluster: failed starting cluster creation "ea-cluster" in resourcegroup="external-auth-cluster-p8xqhgpdsfcg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/external-auth-cluster-p8xqhgpdsfcg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ea-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c4c360>: 
    failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-gxd68nh7r27l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-gxd68nh7r27l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:110]: failed to create HCP cluster negative-tests-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c4c360>: 
    failed to create HCP cluster negative-tests-cluster: failed starting cluster creation "negative-tests-cluster" in resourcegroup="rg-negative-tests-gxd68nh7r27l": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-negative-tests-gxd68nh7r27l/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/negative-tests-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to forward control plane logs to a storage account and Event Hub via shoebox diagnostic settings · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c1e3a0>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-86z97xdsjrzg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-86z97xdsjrzg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/shoebox_logs.go:272]: failed to create HCP cluster shoebox-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c1e3a0>: 
    failed to create HCP cluster shoebox-hcp-cluster: failed starting cluster creation "shoebox-hcp-cluster" in resourcegroup="shoebox-cluster-86z97xdsjrzg": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/shoebox-cluster-86z97xdsjrzg/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/shoebox-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to delete an HCP cluster whose managed identities were deleted first · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc001020160>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-rljtqqgn7sg4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-rljtqqgn7sg4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_delete_missing_identities.go:113]: failed to create HCP cluster "missing-mi-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc001020160>: 
    failed to create HCP cluster missing-mi-hcp-cluster: failed starting cluster creation "missing-mi-hcp-cluster" in resourcegroup="delete-mi-rg-rljtqqgn7sg4": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/delete-mi-rg-rljtqqgn7sg4/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/missing-mi-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc00090c560>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-vh6wflv95g2b": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-vh6wflv95g2b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:113]: failed to create HCP cluster for pull secret test
Unexpected error:
    <*fmt.wrapError | 0xc00090c560>: 
    failed to create HCP cluster pullsecret-hcp-cluster: failed starting cluster creation "pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-vh6wflv95g2b": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/pullsecret-test-vh6wflv95g2b/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/pullsecret-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 5.0 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-x4wgk9" with version 5.0.0-rc.1 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b12240>: 
    failed to create HCP cluster cluster-zstream-5-0-x4wgk9: failed starting cluster creation "cluster-zstream-5-0-x4wgk9" in resourcegroup="rg-zstream-upgrade-5-0-2l8t2pl56plj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-2l8t2pl56plj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-x4wgk9
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-5-0-x4wgk9" with version 5.0.0-rc.1 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000b12240>: 
    failed to create HCP cluster cluster-zstream-5-0-x4wgk9: failed starting cluster creation "cluster-zstream-5-0-x4wgk9" in resourcegroup="rg-zstream-upgrade-5-0-2l8t2pl56plj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-5-0-2l8t2pl56plj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-5-0-x4wgk9
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should upgrade a nodepool skipping one minor version (+2) from 4.20.z to 4.22.zLatest · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-hmzsf6 at version 4.22.14
Unexpected error:
    <*fmt.wrapError | 0xc00011b1c0>: 
    failed to create HCP cluster np-skip-minor-hmzsf6: failed starting cluster creation "np-skip-minor-hmzsf6" in resourcegroup="rg-np-skip-minor-hmzsf6-6wl6vk2jvzqj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-hmzsf6-6wl6vk2jvzqj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-hmzsf6
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:441]: failed to create HCP cluster np-skip-minor-hmzsf6 at version 4.22.14
Unexpected error:
    <*fmt.wrapError | 0xc00011b1c0>: 
    failed to create HCP cluster np-skip-minor-hmzsf6: failed starting cluster creation "np-skip-minor-hmzsf6" in resourcegroup="rg-np-skip-minor-hmzsf6-6wl6vk2jvzqj": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-skip-minor-hmzsf6-6wl6vk2jvzqj/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-skip-minor-hmzsf6
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to cordon, drain, and uncordon a node in an HCP cluster · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c40760>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-zj2gjdls2bft": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-zj2gjdls2bft/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_cordon.go:90]: failed to create HCP cluster cordon-hcp-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000c40760>: 
    failed to create HCP cluster cordon-hcp-cluster: failed starting cluster creation "cordon-hcp-cluster" in resourcegroup="rg-cordon-test-zj2gjdls2bft": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cordon-test-zj2gjdls2bft/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cordon-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to successfully upgrade control plane minor version from 4.20 minor to 4.21 minor · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-jzpfzt" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc0004da3e0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-jzpfzt: failed starting cluster creation "cp-ystream-upgrade-4-21-jzpfzt" in resourcegroup="rg-cp-ystream-upgrade-4-21-dqfxcxkrzpv6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-dqfxcxkrzpv6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-jzpfzt
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_minor_upgrade.go:133]: failed to create HCP cluster "cp-ystream-upgrade-4-21-jzpfzt" at install version 4.20
Unexpected error:
    <*fmt.wrapError | 0xc0004da3e0>: 
    failed to create HCP cluster cp-ystream-upgrade-4-21-jzpfzt: failed starting cluster creation "cp-ystream-upgrade-4-21-jzpfzt" in resourcegroup="rg-cp-ystream-upgrade-4-21-dqfxcxkrzpv6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cp-ystream-upgrade-4-21-dqfxcxkrzpv6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cp-ystream-upgrade-4-21-jzpfzt
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000a3df60>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-zxjm8pgv2pc2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-zxjm8pgv2pc2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:227]: failed to create HCP cluster oidc-wi-cluster
Unexpected error:
    <*fmt.wrapError | 0xc000a3df60>: 
    failed to create HCP cluster oidc-wi-cluster: failed starting cluster creation "oidc-wi-cluster" in resourcegroup="oidc-wi-zxjm8pgv2pc2": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/oidc-wi-zxjm8pgv2pc2/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/oidc-wi-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test ARO-HCP should be able to perform a control plane and node pool install with OCP candidate channel for 4.21 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-pvmhwjpv57ns/cluster-candidate-4-21-bhdxc7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001026100>: 
    failed to create HCP cluster cluster-candidate-4-21-bhdxc7: failed starting cluster creation "cluster-candidate-4-21-bhdxc7" in resourcegroup="rg-candidate-4-21-pvmhwjpv57ns": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-pvmhwjpv57ns/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-21-bhdxc7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create_multiversion.go:122]: HCP cluster rg-candidate-4-21-pvmhwjpv57ns/cluster-candidate-4-21-bhdxc7 should provision
Unexpected error:
    <*fmt.wrapError | 0xc001026100>: 
    failed to create HCP cluster cluster-candidate-4-21-bhdxc7: failed starting cluster creation "cluster-candidate-4-21-bhdxc7" in resourcegroup="rg-candidate-4-21-pvmhwjpv57ns": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-candidate-4-21-pvmhwjpv57ns/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-candidate-4-21-bhdxc7
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Service Provider should upgrade the control plane z-stream automatically on behalf of the customer for 4.20 · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-grlqzp" with version 4.20.37 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000d24180>: 
    failed to create HCP cluster cluster-zstream-4-20-grlqzp: failed starting cluster creation "cluster-zstream-4-20-grlqzp" in resourcegroup="rg-zstream-upgrade-4-20-68d5h45dwbbl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-68d5h45dwbbl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-grlqzp
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:123]: failed to create HCP cluster "cluster-zstream-4-20-grlqzp" with version 4.20.37 on candidate channel
Unexpected error:
    <*fmt.wrapError | 0xc000d24180>: 
    failed to create HCP cluster cluster-zstream-4-20-grlqzp: failed starting cluster creation "cluster-zstream-4-20-grlqzp" in resourcegroup="rg-zstream-upgrade-4-20-68d5h45dwbbl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-zstream-upgrade-4-20-68d5h45dwbbl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/cluster-zstream-4-20-grlqzp
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should create a cluster using v20260901preview API and verify cluster health · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000ff03a0>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-tctrth8t85jl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-tctrth8t85jl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_v20260901preview.go:86]: failed to create HCP cluster "v20260901" via v20260901preview
Unexpected error:
    <*fmt.wrapError | 0xc000ff03a0>: 
    failed to create HCP cluster v20260901: failed starting cluster creation "v20260901" in resourcegroup="v20260901preview-tctrth8t85jl": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/v20260901preview-tctrth8t85jl/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/v20260901
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to issue the default ingress serving certificate from Let's Encrypt using cert-manager with ACME DNS-01 challenges · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000cce600>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-d6fx26fxx7pt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-d6fx26fxx7pt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_ingress_cert_letsencrypt.go:123]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc000cce600>: 
    failed to create HCP cluster ingress-cert-le: failed starting cluster creation "ingress-cert-le" in resourcegroup="ingress-cert-le-d6fx26fxx7pt": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/ingress-cert-le-d6fx26fxx7pt/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/ingress-cert-le
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should create a cluster with private KAS and verify API server is only reachable from VNet while ingress remains public · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000f0c120>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-25q9kvlxrr5f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-25q9kvlxrr5f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kas.go:90]: failed to create HCP cluster "private-kas" with private KAS
Unexpected error:
    <*fmt.wrapError | 0xc000f0c120>: 
    failed to create HCP cluster private-kas: failed starting cluster creation "private-kas" in resourcegroup="private-kas-25q9kvlxrr5f": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/private-kas-25q9kvlxrr5f/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/private-kas
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to list available HCP OpenShift versions and validate response content · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc000acf080>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_versions.go:43]: failed to list HCP OpenShift versions
Unexpected error:
    <*exported.ResponseError | 0xc000acf080>: 
    GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/uksouth/hcpOpenShiftVersions
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00103c1a0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9w46tklt9wd6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9w46tklt9wd6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:88]: failed to create HCP cluster "arm64-vm-hcp-cluster"
Unexpected error:
    <*fmt.wrapError | 0xc00103c1a0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster: failed starting cluster creation "arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-9w46tklt9wd6": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/arm64-vm-cluster-9w46tklt9wd6/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/arm64-vm-hcp-cluster
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
ERROR CODE: BadGatewayConnection; detail message The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request.; provider Microsoft.RedHatOpenShift
02:00:32 UTC · failed at e2e · test Customer should downgrade a nodepool to a lower minor version from 4.22.zLatest to 4.20.zLatest · suite stage/parallel
Show raw failure
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0005a52a0>: 
    failed to create HCP cluster np-dg-minor-7n96xc: failed starting cluster creation "np-dg-minor-7n96xc" in resourcegroup="rg-np-dg-minor-7n96xc-5w7t75k2pb5g": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-7n96xc-5w7t75k2pb5g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-7n96xc
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_version_upgrade.go:720]: failed to create HCP cluster
Unexpected error:
    <*fmt.wrapError | 0xc0005a52a0>: 
    failed to create HCP cluster np-dg-minor-7n96xc: failed starting cluster creation "np-dg-minor-7n96xc" in resourcegroup="rg-np-dg-minor-7n96xc-5w7t75k2pb5g": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-np-dg-minor-7n96xc-5w7t75k2pb5g/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/np-dg-minor-7n96xc
    --------------------------------------------------------------------------------
    RESPONSE 502: 502 Bad Gateway
    ERROR CODE: BadGatewayConnection
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "BadGatewayConnection",
        "message": "The network connectivity issue encountered for 'Microsoft.RedHatOpenShift'; cannot fulfill the request."
      }
    }
    --------------------------------------------------------------------------------
    
    ...
occurred