CIHealth Run Log

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
06:09:05	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	25	e2e (25) timeout during CreateHCPClusterAndWait; context deadline exceeded 06:09:05 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:135]: Unexpected error: <fmt.wrapErrors \| 0xc001606750>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-zrnjn5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-zrnjn5, error: context deadline exceeded { msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-zrnjn5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-zrnjn5, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000821bb0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-zrnjn5", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:135]: Unexpected error: <fmt.wrapErrors \| 0xc001606750>: failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-zrnjn5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-zrnjn5, error: context deadline exceeded { msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-zrnjn5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-zrnjn5, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000821bb0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-zrnjn5", }, <context.deadlineExceededError>{}, ], } occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 06:09:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error: <fmt.wrapErrors \| 0xc000ee85d0>: failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-2qwhkh" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: context deadline exceeded { msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-2qwhkh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0011cf9e0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh", }, <fmt.wrapErrors \| 0xc000ee85a0>{ msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-2qwhkh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0011cf9e0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error: <fmt.wrapErrors \| 0xc000ee85d0>: failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-2qwhkh" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: context deadline exceeded { msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-2qwhkh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0011cf9e0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh", }, <fmt.wrapErrors \| 0xc000ee85a0>{ msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-2qwhkh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0011cf9e0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh", }, <context.deadlineExceededError>{}, ], }, ], } occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 06:09:05 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:117]: Unexpected error: <fmt.wrapError \| 0xc000e16f40>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-5p4qlp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a", "name": "cfc5391b-7b48-4657-9b50-67b79747ab7a", "status": "Failed", "startTime": "2026-03-30T06:22:25.344609849Z", "endTime": "2026-03-30T06:22:55.779876405Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5p4qlp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a\",\n \"name\": \"cfc5391b-7b48-4657-9b50-67b79747ab7a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:22:25.344609849Z\",\n \"endTime\": \"2026-03-30T06:22:55.779876405Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e16f20>{ msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5p4qlp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a\",\n \"name\": \"cfc5391b-7b48-4657-9b50-67b79747ab7a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:22:25.344609849Z\",\n \"endTime\": \"2026-03-30T06:22:55.779876405Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000fe3860>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Content-Length": ["503"], "X-Ms-Request-Id": [ "141c0618-2615-4e89-98c8-24cc9f4c8553", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Correlation-Request-Id": [ "f073cfd3-865e-46e9-9354-23fc2d617e27", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T062256Z:f073cfd3-865e-46e9-9354-23fc2d617e27", ], "Expires": ["-1"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Msedge-Ref": [ "Ref A: CDC0382112E64ECDBEECE2C1D1A01FE0 Ref B: CH1AA2020617025 Ref C: 2026-03-30T06:22:56Z", ], "Content-Type": ["application/json"], "X-Envoy-Upstream-Service-Time": ["9"], "X-Content-Type-Options": ["nosniff"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:117]: Unexpected error: <fmt.wrapError \| 0xc000e16f40>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-5p4qlp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a", "name": "cfc5391b-7b48-4657-9b50-67b79747ab7a", "status": "Failed", "startTime": "2026-03-30T06:22:25.344609849Z", "endTime": "2026-03-30T06:22:55.779876405Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5p4qlp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a\",\n \"name\": \"cfc5391b-7b48-4657-9b50-67b79747ab7a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:22:25.344609849Z\",\n \"endTime\": \"2026-03-30T06:22:55.779876405Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e16f20>{ msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5p4qlp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a\",\n \"name\": \"cfc5391b-7b48-4657-9b50-67b79747ab7a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:22:25.344609849Z\",\n \"endTime\": \"2026-03-30T06:22:55.779876405Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000fe3860>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Content-Length": ["503"], "X-Ms-Request-Id": [ "141c0618-2615-4e89-98c8-24cc9f4c8553", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Correlation-Request-Id": [ "f073cfd3-865e-46e9-9354-23fc2d617e27", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T062256Z:f073cfd3-865e-46e9-9354-23fc2d617e27", ], "Expires": ["-1"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Msedge-Ref": [ "Ref A: CDC0382112E64ECDBEECE2C1D1A01FE0 Ref B: CH1AA2020617025 Ref C: 2026-03-30T06:22:56Z", ], "Content-Type": ["application/json"], "X-Envoy-Upstream-Service-Time": ["9"], "X-Content-Type-Options": ["nosniff"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 06:09:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error: <fmt.wrapErrors \| 0xc00148d140>: failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-mww9kq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: context deadline exceeded { msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-mww9kq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0012346f0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq", }, <fmt.wrapErrors \| 0xc00148d0e0>{ msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-mww9kq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0012346f0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error: <fmt.wrapErrors \| 0xc00148d140>: failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-mww9kq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: context deadline exceeded { msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-mww9kq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0012346f0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq", }, <fmt.wrapErrors \| 0xc00148d0e0>{ msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-mww9kq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0012346f0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq", }, <context.deadlineExceededError>{}, ], }, ], } occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 06:09:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error: <fmt.wrapError \| 0xc000c50ee0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-n5nffq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b", "name": "1517a95b-a81b-4ce8-841d-22153214295b", "status": "Failed", "startTime": "2026-03-30T06:15:55.940417137Z", "endTime": "2026-03-30T06:16:38.146603108Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n5nffq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b\",\n \"name\": \"1517a95b-a81b-4ce8-841d-22153214295b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:15:55.940417137Z\",\n \"endTime\": \"2026-03-30T06:16:38.146603108Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000c50ec0>{ msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n5nffq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b\",\n \"name\": \"1517a95b-a81b-4ce8-841d-22153214295b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:15:55.940417137Z\",\n \"endTime\": \"2026-03-30T06:16:38.146603108Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0004c5650>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Content-Length": ["503"], "Expires": ["-1"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Content-Type-Options": ["nosniff"], "Date": [ "Mon, 30 Mar 2026 06:16:46 GMT", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/28c44e1a-d416-41d6-a0fc-92aed87244cd", ], "X-Ms-Correlation-Request-Id": [ "ec54b0dc-0811-4ecd-92b7-fd2310698c78", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T061647Z:ec54b0dc-0811-4ecd-92b7-fd2310698c78", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error: <fmt.wrapError \| 0xc000c50ee0>: failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-n5nffq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b", "name": "1517a95b-a81b-4ce8-841d-22153214295b", "status": "Failed", "startTime": "2026-03-30T06:15:55.940417137Z", "endTime": "2026-03-30T06:16:38.146603108Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n5nffq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b\",\n \"name\": \"1517a95b-a81b-4ce8-841d-22153214295b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:15:55.940417137Z\",\n \"endTime\": \"2026-03-30T06:16:38.146603108Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000c50ec0>{ msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n5nffq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b\",\n \"name\": \"1517a95b-a81b-4ce8-841d-22153214295b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:15:55.940417137Z\",\n \"endTime\": \"2026-03-30T06:16:38.146603108Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0004c5650>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Content-Length": ["503"], "Expires": ["-1"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Content-Type-Options": ["nosniff"], "Date": [ "Mon, 30 Mar 2026 06:16:46 GMT", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/28c44e1a-d416-41d6-a0fc-92aed87244cd", ], "X-Ms-Correlation-Request-Id": [ "ec54b0dc-0811-4ecd-92b7-fd2310698c78", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T061647Z:ec54b0dc-0811-4ecd-92b7-fd2310698c78", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 06:09:05 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error: <fmt.wrapErrors \| 0xc00165cd20>: failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-7cfr2p" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: context deadline exceeded { msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-7cfr2p\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000e18a90>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p", }, <fmt.wrapErrors \| 0xc00165ccf0>{ msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-7cfr2p\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000e18a90>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error: <fmt.wrapErrors \| 0xc00165cd20>: failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-7cfr2p" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: context deadline exceeded { msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-7cfr2p\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000e18a90>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p", }, <fmt.wrapErrors \| 0xc00165ccf0>{ msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-7cfr2p\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000e18a90>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p", }, <context.deadlineExceededError>{}, ], }, ], } occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 06:09:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error: <fmt.wrapErrors \| 0xc001198d50>: failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-rqfwmq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: context deadline exceeded { msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rqfwmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00111ad80>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq", }, <fmt.wrapErrors \| 0xc001198d20>{ msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rqfwmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00111ad80>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error: <fmt.wrapErrors \| 0xc001198d50>: failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-rqfwmq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: context deadline exceeded { msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rqfwmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00111ad80>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq", }, <fmt.wrapErrors \| 0xc001198d20>{ msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rqfwmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00111ad80>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq", }, <context.deadlineExceededError>{}, ], }, ], } occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 06:09:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapError \| 0xc000e0b7c0>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-l2s4f2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02", "name": "f9df063e-f124-4f77-8a25-358c9d5cde02", "status": "Failed", "startTime": "2026-03-30T06:16:32.010035382Z", "endTime": "2026-03-30T06:16:48.681254232Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-l2s4f2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\",\n \"name\": \"f9df063e-f124-4f77-8a25-358c9d5cde02\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:16:32.010035382Z\",\n \"endTime\": \"2026-03-30T06:16:48.681254232Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e0b7a0>{ msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-l2s4f2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\",\n \"name\": \"f9df063e-f124-4f77-8a25-358c9d5cde02\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:16:32.010035382Z\",\n \"endTime\": \"2026-03-30T06:16:48.681254232Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00104d650>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T061652Z:5fa8b3d2-fffd-493a-9339-7cb51375d257", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "Content-Length": ["503"], "X-Envoy-Upstream-Service-Time": ["9"], "Date": [ "Mon, 30 Mar 2026 06:16:51 GMT", ], "Expires": ["-1"], "X-Msedge-Ref": [ "Ref A: 9C0B547CA14B41D4A16FDC03FE815989 Ref B: CHI261140514036 Ref C: 2026-03-30T06:16:52Z", ], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapError \| 0xc000e0b7c0>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-l2s4f2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02", "name": "f9df063e-f124-4f77-8a25-358c9d5cde02", "status": "Failed", "startTime": "2026-03-30T06:16:32.010035382Z", "endTime": "2026-03-30T06:16:48.681254232Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-l2s4f2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\",\n \"name\": \"f9df063e-f124-4f77-8a25-358c9d5cde02\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:16:32.010035382Z\",\n \"endTime\": \"2026-03-30T06:16:48.681254232Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000e0b7a0>{ msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-l2s4f2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\",\n \"name\": \"f9df063e-f124-4f77-8a25-358c9d5cde02\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:16:32.010035382Z\",\n \"endTime\": \"2026-03-30T06:16:48.681254232Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00104d650>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T061652Z:5fa8b3d2-fffd-493a-9339-7cb51375d257", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "Content-Length": ["503"], "X-Envoy-Upstream-Service-Time": ["9"], "Date": [ "Mon, 30 Mar 2026 06:16:51 GMT", ], "Expires": ["-1"], "X-Msedge-Ref": [ "Ref A: 9C0B547CA14B41D4A16FDC03FE815989 Ref B: CHI261140514036 Ref C: 2026-03-30T06:16:52Z", ], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 06:09:05 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error: <fmt.wrapError \| 0xc00082f8a0>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-sx6f6k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4", "name": "75078195-e41b-452a-877c-5d9b8e76e3e4", "status": "Failed", "startTime": "2026-03-30T06:20:53.192418254Z", "endTime": "2026-03-30T06:21:52.368868636Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-sx6f6k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4\",\n \"name\": \"75078195-e41b-452a-877c-5d9b8e76e3e4\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:20:53.192418254Z\",\n \"endTime\": \"2026-03-30T06:21:52.368868636Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00082f880>{ msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-sx6f6k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4\",\n \"name\": \"75078195-e41b-452a-877c-5d9b8e76e3e4\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:20:53.192418254Z\",\n \"endTime\": \"2026-03-30T06:21:52.368868636Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0014c9ec0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "Cache-Control": ["no-cache"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "6a949a5b-9896-4995-9b24-95e48c1486b0", ], "X-Envoy-Upstream-Service-Time": ["19"], "Pragma": ["no-cache"], "Content-Length": ["503"], "Expires": ["-1"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T062154Z:51754cc2-4bfb-40ab-9d75-1f562f630654", ], "Date": [ "Mon, 30 Mar 2026 06:21:54 GMT", ], "X-Ms-Operation-Identifier": [ "appId=c94dda6b-80b4-4144-a102-590... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error: <fmt.wrapError \| 0xc00082f8a0>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-sx6f6k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4", "name": "75078195-e41b-452a-877c-5d9b8e76e3e4", "status": "Failed", "startTime": "2026-03-30T06:20:53.192418254Z", "endTime": "2026-03-30T06:21:52.368868636Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-sx6f6k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4\",\n \"name\": \"75078195-e41b-452a-877c-5d9b8e76e3e4\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:20:53.192418254Z\",\n \"endTime\": \"2026-03-30T06:21:52.368868636Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00082f880>{ msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-sx6f6k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4\",\n \"name\": \"75078195-e41b-452a-877c-5d9b8e76e3e4\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:20:53.192418254Z\",\n \"endTime\": \"2026-03-30T06:21:52.368868636Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0014c9ec0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "Cache-Control": ["no-cache"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "6a949a5b-9896-4995-9b24-95e48c1486b0", ], "X-Envoy-Upstream-Service-Time": ["19"], "Pragma": ["no-cache"], "Content-Length": ["503"], "Expires": ["-1"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T062154Z:51754cc2-4bfb-40ab-9d75-1f562f630654", ], "Date": [ "Mon, 30 Mar 2026 06:21:54 GMT", ], "X-Ms-Operation-Identifier": [ "appId=c94dda6b-80b4-4144-a102-590... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred Cluster provisioning failed 06:09:05 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:153]: Cluster provisioning failed fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:153]: Cluster provisioning failed timeout during CreateHCPClusterAndWait; context deadline exceeded 06:09:05 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error: <fmt.wrapErrors \| 0xc001454c30>: failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-528g9r" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: context deadline exceeded { msg: "failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-528g9r\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000e7cad0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r", }, <fmt.wrapErrors \| 0xc001454c00>{ msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-528g9r\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000e7cad0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error: <fmt.wrapErrors \| 0xc001454c30>: failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-528g9r" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: context deadline exceeded { msg: "failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-528g9r\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000e7cad0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r", }, <fmt.wrapErrors \| 0xc001454c00>{ msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-528g9r\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000e7cad0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r", }, <context.deadlineExceededError>{}, ], }, ], } occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 06:09:05 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error: <fmt.wrapError \| 0xc00099c3e0>: failed to create HCP cluster ea-listdjh5pl: failed waiting for cluster="ea-listdjh5pl" in resourcegroup="ea-list-vc9fmh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41", "name": "d92e5056-78d6-404e-a6ae-95d01a203d41", "status": "Failed", "startTime": "2026-03-30T06:28:34.388150903Z", "endTime": "2026-03-30T06:29:01.410266778Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-listdjh5pl: failed waiting for cluster=\"ea-listdjh5pl\" in resourcegroup=\"ea-list-vc9fmh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41\",\n \"name\": \"d92e5056-78d6-404e-a6ae-95d01a203d41\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:28:34.388150903Z\",\n \"endTime\": \"2026-03-30T06:29:01.410266778Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00099c3c0>{ msg: "failed waiting for cluster=\"ea-listdjh5pl\" in resourcegroup=\"ea-list-vc9fmh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41\",\n \"name\": \"d92e5056-78d6-404e-a6ae-95d01a203d41\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:28:34.388150903Z\",\n \"endTime\": \"2026-03-30T06:29:01.410266778Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00128a3f0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["503"], "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Correlation-Request-Id": [ "10073257-81d6-4a5d-9336-e607cce74141", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Pragma": ["no-cache"], "Expires": ["-1"], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: 93FDCB2B04AE464DBCEFBE7C2F6542A0 Ref B: CHI261140502034 Ref C: 2026-03-30T06:29:05Z", ], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "3d7e90c1-6210-4595-a613-3eb6e9018232", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96db... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error: <fmt.wrapError \| 0xc00099c3e0>: failed to create HCP cluster ea-listdjh5pl: failed waiting for cluster="ea-listdjh5pl" in resourcegroup="ea-list-vc9fmh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41", "name": "d92e5056-78d6-404e-a6ae-95d01a203d41", "status": "Failed", "startTime": "2026-03-30T06:28:34.388150903Z", "endTime": "2026-03-30T06:29:01.410266778Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-listdjh5pl: failed waiting for cluster=\"ea-listdjh5pl\" in resourcegroup=\"ea-list-vc9fmh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41\",\n \"name\": \"d92e5056-78d6-404e-a6ae-95d01a203d41\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:28:34.388150903Z\",\n \"endTime\": \"2026-03-30T06:29:01.410266778Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00099c3c0>{ msg: "failed waiting for cluster=\"ea-listdjh5pl\" in resourcegroup=\"ea-list-vc9fmh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41\",\n \"name\": \"d92e5056-78d6-404e-a6ae-95d01a203d41\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:28:34.388150903Z\",\n \"endTime\": \"2026-03-30T06:29:01.410266778Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00128a3f0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["503"], "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Correlation-Request-Id": [ "10073257-81d6-4a5d-9336-e607cce74141", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Pragma": ["no-cache"], "Expires": ["-1"], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: 93FDCB2B04AE464DBCEFBE7C2F6542A0 Ref B: CHI261140502034 Ref C: 2026-03-30T06:29:05Z", ], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "3d7e90c1-6210-4595-a613-3eb6e9018232", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96db... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 06:09:05 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error: <fmt.wrapError \| 0xc00089a340>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-gsvbfn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8", "name": "44da3a3a-6183-4e59-9edc-5597fe1650b8", "status": "Failed", "startTime": "2026-03-30T06:20:01.753593452Z", "endTime": "2026-03-30T06:20:48.88837796Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ephemeral-disk: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-gsvbfn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8\",\n \"name\": \"44da3a3a-6183-4e59-9edc-5597fe1650b8\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:20:01.753593452Z\",\n \"endTime\": \"2026-03-30T06:20:48.88837796Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00089a320>{ msg: "failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-gsvbfn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8\",\n \"name\": \"44da3a3a-6183-4e59-9edc-5597fe1650b8\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:20:01.753593452Z\",\n \"endTime\": \"2026-03-30T06:20:48.88837796Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0008641b0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Content-Length": ["502"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"], "Date": [ "Mon, 30 Mar 2026 06:20:52 GMT", ], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "a3b37884-de6d-4b7e-bb88-9da4257ef455", ], "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Correlation-Request-Id": [ "eb37ef8b-2459-4a5c-9a4e-c2ad42298d82", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T062052Z:eb37ef8b-2459-4a5c-9a4e-c2ad42298d82", ], "X-Cache": ["CONFIG_NOCACHE"], "Pragma": ["no-cache"], "Expires": ["-1"], "X-Msedge-Ref": [ "Ref A: 17B0B8CD7B0E4D9BB4BC8B91EFCA7646 Ref B: CH1AA2020617019 Ref C: 2026-03-30T06:20:52Z... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error: <fmt.wrapError \| 0xc00089a340>: failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-gsvbfn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8", "name": "44da3a3a-6183-4e59-9edc-5597fe1650b8", "status": "Failed", "startTime": "2026-03-30T06:20:01.753593452Z", "endTime": "2026-03-30T06:20:48.88837796Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ephemeral-disk: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-gsvbfn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8\",\n \"name\": \"44da3a3a-6183-4e59-9edc-5597fe1650b8\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:20:01.753593452Z\",\n \"endTime\": \"2026-03-30T06:20:48.88837796Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00089a320>{ msg: "failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-gsvbfn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8\",\n \"name\": \"44da3a3a-6183-4e59-9edc-5597fe1650b8\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:20:01.753593452Z\",\n \"endTime\": \"2026-03-30T06:20:48.88837796Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0008641b0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Content-Length": ["502"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"], "Date": [ "Mon, 30 Mar 2026 06:20:52 GMT", ], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "a3b37884-de6d-4b7e-bb88-9da4257ef455", ], "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Correlation-Request-Id": [ "eb37ef8b-2459-4a5c-9a4e-c2ad42298d82", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T062052Z:eb37ef8b-2459-4a5c-9a4e-c2ad42298d82", ], "X-Cache": ["CONFIG_NOCACHE"], "Pragma": ["no-cache"], "Expires": ["-1"], "X-Msedge-Ref": [ "Ref A: 17B0B8CD7B0E4D9BB4BC8B91EFCA7646 Ref B: CH1AA2020617019 Ref C: 2026-03-30T06:20:52Z... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network 06:09:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error: <fmt.wrapError \| 0xc000334400>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-57zxqg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31", "name": "3c3b1341-fda3-41e9-8dd4-5c30342a2d31", "status": "Failed", "startTime": "2026-03-30T06:12:39.328826001Z", "endTime": "2026-03-30T06:13:08.581005823Z", "error": { "code": "InternalServerError", "message": "Failed to get virtual network 'customer-vnet-name-4-19' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client '3820d679-b809-47f8-8f80-79cbd25d1d89' with object id '62b94c2a-37fd-4b0a-b97c-1370b69f6c0c' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-57zxqg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31\",\n \"name\": \"3c3b1341-fda3-41e9-8dd4-5c30342a2d31\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:12:39.328826001Z\",\n \"endTime\": \"2026-03-30T06:13:08.581005823Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-name-4-19' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client '3820d679-b809-47f8-8f80-79cbd25d1d89' with object id '62b94c2a-37fd-4b0a-b97c-1370b69f6c0c' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001475530>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Pragma": ["no-cache"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/f58e194a-3f86-4008-98dc-aa5f60802195", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: 60DFA7F6143244D1924498FA54547EC7 Ref B: CHI261140510034 Ref C: 2026-03-30T06:13:10Z", ], "Date": [ "Mon, 30 Mar 2026 06:13:10 GMT", ], "Cache-Control": ["no-cache"], "Expires": ["-1"], "X-Ms-Correlation-Request-Id": [ "bdec501d-0ca0-4486-8500-77afaa68e37a", ], "X-Ms-Routing-Request-Id": [ "CANADAEAST:20260330T061310Z:bdec501d-0ca0-4486-8500-77afaa68e37a", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Envoy-Upstream-Service-Time": ["9"], "Content-Length": ["1563"], "Content-Type": ["application/json"], "X-M... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error: <fmt.wrapError \| 0xc000334400>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-57zxqg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31", "name": "3c3b1341-fda3-41e9-8dd4-5c30342a2d31", "status": "Failed", "startTime": "2026-03-30T06:12:39.328826001Z", "endTime": "2026-03-30T06:13:08.581005823Z", "error": { "code": "InternalServerError", "message": "Failed to get virtual network 'customer-vnet-name-4-19' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client '3820d679-b809-47f8-8f80-79cbd25d1d89' with object id '62b94c2a-37fd-4b0a-b97c-1370b69f6c0c' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n" } } -------------------------------------------------------------------------------- { msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-57zxqg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31\",\n \"name\": \"3c3b1341-fda3-41e9-8dd4-5c30342a2d31\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:12:39.328826001Z\",\n \"endTime\": \"2026-03-30T06:13:08.581005823Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-name-4-19' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n \\\"error\\\": {\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\n \\\"message\\\": \\\"The client '3820d679-b809-47f8-8f80-79cbd25d1d89' with object id '62b94c2a-37fd-4b0a-b97c-1370b69f6c0c' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n }\\n}\\n--------------------------------------------------------------------------------\\n\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001475530>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Pragma": ["no-cache"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/f58e194a-3f86-4008-98dc-aa5f60802195", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: 60DFA7F6143244D1924498FA54547EC7 Ref B: CHI261140510034 Ref C: 2026-03-30T06:13:10Z", ], "Date": [ "Mon, 30 Mar 2026 06:13:10 GMT", ], "Cache-Control": ["no-cache"], "Expires": ["-1"], "X-Ms-Correlation-Request-Id": [ "bdec501d-0ca0-4486-8500-77afaa68e37a", ], "X-Ms-Routing-Request-Id": [ "CANADAEAST:20260330T061310Z:bdec501d-0ca0-4486-8500-77afaa68e37a", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Envoy-Upstream-Service-Time": ["9"], "Content-Length": ["1563"], "Content-Type": ["application/json"], "X-M... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 06:09:05 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: Unexpected error: <fmt.wrapError \| 0xc000a0b040>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-t4849k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395", "name": "671489be-ab4c-40e2-81b0-625f59cf6395", "status": "Failed", "startTime": "2026-03-30T06:22:24.890658519Z", "endTime": "2026-03-30T06:22:55.769894295Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-t4849k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\",\n \"name\": \"671489be-ab4c-40e2-81b0-625f59cf6395\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:22:24.890658519Z\",\n \"endTime\": \"2026-03-30T06:22:55.769894295Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a0b020>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-t4849k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\",\n \"name\": \"671489be-ab4c-40e2-81b0-625f59cf6395\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:22:24.890658519Z\",\n \"endTime\": \"2026-03-30T06:22:55.769894295Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0011f0ed0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/be562cea-8623-4678-adaa-0057ff53bbd9", ], "X-Ms-Correlation-Request-Id": [ "8b2df379-7ba3-4320-8e92-21a566e40b6c", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Msedge-Ref": [ "Ref A: BC420E598D454882AFE3F4518AA17A00 Ref B: CHI261140512052 Ref C: 2026-03-30T06:22:55Z", ], "Content-Length": ["503"], "X-Ms-Request-Id": [ "fef2e839-309a-4189-8dfd-f18f57fe8ccf", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Ratelimit-Rema... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: Unexpected error: <fmt.wrapError \| 0xc000a0b040>: failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-t4849k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395", "name": "671489be-ab4c-40e2-81b0-625f59cf6395", "status": "Failed", "startTime": "2026-03-30T06:22:24.890658519Z", "endTime": "2026-03-30T06:22:55.769894295Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-t4849k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\",\n \"name\": \"671489be-ab4c-40e2-81b0-625f59cf6395\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:22:24.890658519Z\",\n \"endTime\": \"2026-03-30T06:22:55.769894295Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a0b020>{ msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-t4849k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\",\n \"name\": \"671489be-ab4c-40e2-81b0-625f59cf6395\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:22:24.890658519Z\",\n \"endTime\": \"2026-03-30T06:22:55.769894295Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0011f0ed0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/be562cea-8623-4678-adaa-0057ff53bbd9", ], "X-Ms-Correlation-Request-Id": [ "8b2df379-7ba3-4320-8e92-21a566e40b6c", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Msedge-Ref": [ "Ref A: BC420E598D454882AFE3F4518AA17A00 Ref B: CHI261140512052 Ref C: 2026-03-30T06:22:55Z", ], "Content-Length": ["503"], "X-Ms-Request-Id": [ "fef2e839-309a-4189-8dfd-f18f57fe8ccf", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Ratelimit-Rema... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 06:09:05 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapError \| 0xc00130dda0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-zwkrrf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a", "name": "f32c7b79-2d68-46c9-a087-244c9417fc9a", "status": "Failed", "startTime": "2026-03-30T06:26:33.891991698Z", "endTime": "2026-03-30T06:27:03.911904895Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-zwkrrf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a\",\n \"name\": \"f32c7b79-2d68-46c9-a087-244c9417fc9a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:26:33.891991698Z\",\n \"endTime\": \"2026-03-30T06:27:03.911904895Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00130dd80>{ msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-zwkrrf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a\",\n \"name\": \"f32c7b79-2d68-46c9-a087-244c9417fc9a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:26:33.891991698Z\",\n \"endTime\": \"2026-03-30T06:27:03.911904895Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000d77380>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Content-Type-Options": ["nosniff"], "Date": [ "Mon, 30 Mar 2026 06:27:04 GMT", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/8159515f-155b-4e28-bb52-27a342aff475", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T062704Z:0a3e91fb-bed7-4520-9179-de2ded4f9ad5", ], "X-Cache": ["CONFIG_NOCACHE"], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Content-Type": ["application/json"], "Expires": ["-1"], "X-Ms-Request-Id": [ "cb801ab9-8e3a-4550-9150-d3ab4ebedcb1", ], "Strict-Transport-Security": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapError \| 0xc00130dda0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-zwkrrf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a", "name": "f32c7b79-2d68-46c9-a087-244c9417fc9a", "status": "Failed", "startTime": "2026-03-30T06:26:33.891991698Z", "endTime": "2026-03-30T06:27:03.911904895Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-zwkrrf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a\",\n \"name\": \"f32c7b79-2d68-46c9-a087-244c9417fc9a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:26:33.891991698Z\",\n \"endTime\": \"2026-03-30T06:27:03.911904895Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00130dd80>{ msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-zwkrrf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a\",\n \"name\": \"f32c7b79-2d68-46c9-a087-244c9417fc9a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:26:33.891991698Z\",\n \"endTime\": \"2026-03-30T06:27:03.911904895Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000d77380>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Content-Type-Options": ["nosniff"], "Date": [ "Mon, 30 Mar 2026 06:27:04 GMT", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/8159515f-155b-4e28-bb52-27a342aff475", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T062704Z:0a3e91fb-bed7-4520-9179-de2ded4f9ad5", ], "X-Cache": ["CONFIG_NOCACHE"], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Content-Type": ["application/json"], "Expires": ["-1"], "X-Ms-Request-Id": [ "cb801ab9-8e3a-4550-9150-d3ab4ebedcb1", ], "Strict-Transport-Security": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred Interrupted by User 06:09:05 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 06:09:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error: <fmt.wrapError \| 0xc000c0e020>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-q6f5dn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a", "name": "ddad76d1-5b78-4fcc-8952-2fc6b191098a", "status": "Failed", "startTime": "2026-03-30T06:24:34.053136626Z", "endTime": "2026-03-30T06:25:20.835964185Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-q6f5dn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a\",\n \"name\": \"ddad76d1-5b78-4fcc-8952-2fc6b191098a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:24:34.053136626Z\",\n \"endTime\": \"2026-03-30T06:25:20.835964185Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000c0e000>{ msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-q6f5dn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a\",\n \"name\": \"ddad76d1-5b78-4fcc-8952-2fc6b191098a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:24:34.053136626Z\",\n \"endTime\": \"2026-03-30T06:25:20.835964185Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0011a6f90>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "25de98a2-737b-4e00-a318-ad55c3f7864f", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Content-Length": ["503"], "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/476c285c-9787-44a3-96f6-2cde38d0c586", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Content-Type": ["application/json"], "Expires": ["-1"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T062525Z:b4a6cd2c-a7ff-... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error: <fmt.wrapError \| 0xc000c0e020>: failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-q6f5dn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a", "name": "ddad76d1-5b78-4fcc-8952-2fc6b191098a", "status": "Failed", "startTime": "2026-03-30T06:24:34.053136626Z", "endTime": "2026-03-30T06:25:20.835964185Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-q6f5dn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a\",\n \"name\": \"ddad76d1-5b78-4fcc-8952-2fc6b191098a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:24:34.053136626Z\",\n \"endTime\": \"2026-03-30T06:25:20.835964185Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000c0e000>{ msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-q6f5dn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a\",\n \"name\": \"ddad76d1-5b78-4fcc-8952-2fc6b191098a\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:24:34.053136626Z\",\n \"endTime\": \"2026-03-30T06:25:20.835964185Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0011a6f90>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "25de98a2-737b-4e00-a318-ad55c3f7864f", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Content-Length": ["503"], "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/476c285c-9787-44a3-96f6-2cde38d0c586", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Content-Type": ["application/json"], "Expires": ["-1"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T062525Z:b4a6cd2c-a7ff-... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 06:09:05 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:100]: Unexpected error: <fmt.wrapError \| 0xc000c03700>: failed to create HCP cluster cluster-zstream-4-20-8kr6g4: failed waiting for cluster="cluster-zstream-4-20-8kr6g4" in resourcegroup="rg-zstream-upgrade-4-20-m22d4f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569", "name": "6855e468-1276-4eca-9790-907f9d976569", "status": "Failed", "startTime": "2026-03-30T06:25:33.698883153Z", "endTime": "2026-03-30T06:26:01.513394715Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-8kr6g4: failed waiting for cluster=\"cluster-zstream-4-20-8kr6g4\" in resourcegroup=\"rg-zstream-upgrade-4-20-m22d4f\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569\",\n \"name\": \"6855e468-1276-4eca-9790-907f9d976569\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:25:33.698883153Z\",\n \"endTime\": \"2026-03-30T06:26:01.513394715Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000c036e0>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-8kr6g4\" in resourcegroup=\"rg-zstream-upgrade-4-20-m22d4f\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569\",\n \"name\": \"6855e468-1276-4eca-9790-907f9d976569\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:25:33.698883153Z\",\n \"endTime\": \"2026-03-30T06:26:01.513394715Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0013690e0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Correlation-Request-Id": [ "11f4df43-a99c-4d0b-85cc-765a6c02b47f", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T062604Z:11f4df43-a99c-4d0b-85cc-765a6c02b47f", ], "X-Msedge-Ref": [ "Ref A: D55D7FFF469D4C5DA1B7711B27E2352B Ref B: CHI261140509025 Ref C: 2026-03-30T06:26:04Z", ], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Ms-Request-Id": [ "4544b9fd-9af5-4ef6-ad11-8a7cc26e6366", ], "X-Envoy-Upstream-Service-Time": ["11"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "Content-Le... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:100]: Unexpected error: <fmt.wrapError \| 0xc000c03700>: failed to create HCP cluster cluster-zstream-4-20-8kr6g4: failed waiting for cluster="cluster-zstream-4-20-8kr6g4" in resourcegroup="rg-zstream-upgrade-4-20-m22d4f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569", "name": "6855e468-1276-4eca-9790-907f9d976569", "status": "Failed", "startTime": "2026-03-30T06:25:33.698883153Z", "endTime": "2026-03-30T06:26:01.513394715Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-8kr6g4: failed waiting for cluster=\"cluster-zstream-4-20-8kr6g4\" in resourcegroup=\"rg-zstream-upgrade-4-20-m22d4f\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569\",\n \"name\": \"6855e468-1276-4eca-9790-907f9d976569\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:25:33.698883153Z\",\n \"endTime\": \"2026-03-30T06:26:01.513394715Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000c036e0>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-8kr6g4\" in resourcegroup=\"rg-zstream-upgrade-4-20-m22d4f\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569\",\n \"name\": \"6855e468-1276-4eca-9790-907f9d976569\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:25:33.698883153Z\",\n \"endTime\": \"2026-03-30T06:26:01.513394715Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0013690e0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Correlation-Request-Id": [ "11f4df43-a99c-4d0b-85cc-765a6c02b47f", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T062604Z:11f4df43-a99c-4d0b-85cc-765a6c02b47f", ], "X-Msedge-Ref": [ "Ref A: D55D7FFF469D4C5DA1B7711B27E2352B Ref B: CHI261140509025 Ref C: 2026-03-30T06:26:04Z", ], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Ms-Request-Id": [ "4544b9fd-9af5-4ef6-ad11-8a7cc26e6366", ], "X-Envoy-Upstream-Service-Time": ["11"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "Content-Le... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred failed creating deployment "test-vm" in resourcegroup="<resource-group>": PUT <url> 06:09:05 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:114]: Timed out after 1800.001s. Expected success, but got an error: <fmt.wrapError \| 0xc0012c8180>: failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-s5czh5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-s5czh5/providers/Microsoft.Resources/deployments/test-vm -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidTemplateDeployment -------------------------------------------------------------------------------- { "error": { "code": "InvalidTemplateDeployment", "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.", "details": [ { "code": "ResourceCountExceedsLimitDueToTemplate", "message": "Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress", "details": [] } ] } } -------------------------------------------------------------------------------- { msg: "failed creating deployment \"test-vm\" in resourcegroup=\"e2e-cidr-connectivity-s5czh5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-s5czh5/providers/Microsoft.Resources/deployments/test-vm\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplateDeployment\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidTemplateDeployment\",\n \"message\": \"The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.\",\n \"details\": [\n {\n \"code\": \"ResourceCountExceedsLimitDueToTemplate\",\n \"message\": \"Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress\",\n \"details\": []\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000f127e0>{ ErrorCode: "InvalidTemplateDeployment", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["198"], "X-Ms-Failure-Cause": ["gateway"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2998"], "X-Ms-Request-Id": [ "cf5d91a3-a913-4d5b-9164-bccf81eb8eee", ], "X-Ms-Correlation-Request-Id": [ "cf5d91a3-a913-4d5b-9164-bccf81eb8eee", ], "X-Ms-Routing-Request-Id": [ "EASTUS2EUAP:20260330T065415Z:cf5d91a3-a913-4d5b-9164-bccf81eb8eee", ], "X-Content-Type-Options": ["nosniff"], "Date": [ "Mon, 30 Mar 2026 06:54:14 GMT", ], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Expires": ["-1"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Msedge-Ref": [ "Ref A: FED97ACCAC0E49C1AB5718A871C140C7 Ref B: CHI261140514023 Ref C: 2026-03-30T06:54:14Z", ], "Content-Length": ["655"], "Content-Type": [ "application/json; charset=utf-8", ], "X-Cache": ["CONFIG_NOCACHE"], }, Body: <exported.nopClosingBytesReader \| 0xc0012c8040>{ s: "{\"error\":{\"code\":\"InvalidTemplateDeployment\",\"message\":\"The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.\",\"details\":[{\"code\":\"ResourceCountExceedsLimitDueToTemplate\",\"message\":\"Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress\",\"detai... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:114]: Timed out after 1800.001s. Expected success, but got an error: <fmt.wrapError \| 0xc0012c8180>: failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-s5czh5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-s5czh5/providers/Microsoft.Resources/deployments/test-vm -------------------------------------------------------------------------------- RESPONSE 400: 400 Bad Request ERROR CODE: InvalidTemplateDeployment -------------------------------------------------------------------------------- { "error": { "code": "InvalidTemplateDeployment", "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.", "details": [ { "code": "ResourceCountExceedsLimitDueToTemplate", "message": "Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress", "details": [] } ] } } -------------------------------------------------------------------------------- { msg: "failed creating deployment \"test-vm\" in resourcegroup=\"e2e-cidr-connectivity-s5czh5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-s5czh5/providers/Microsoft.Resources/deployments/test-vm\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplateDeployment\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"InvalidTemplateDeployment\",\n \"message\": \"The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.\",\n \"details\": [\n {\n \"code\": \"ResourceCountExceedsLimitDueToTemplate\",\n \"message\": \"Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress\",\n \"details\": []\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000f127e0>{ ErrorCode: "InvalidTemplateDeployment", StatusCode: 400, RawResponse: { Status: "400 Bad Request", StatusCode: 400, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["198"], "X-Ms-Failure-Cause": ["gateway"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2998"], "X-Ms-Request-Id": [ "cf5d91a3-a913-4d5b-9164-bccf81eb8eee", ], "X-Ms-Correlation-Request-Id": [ "cf5d91a3-a913-4d5b-9164-bccf81eb8eee", ], "X-Ms-Routing-Request-Id": [ "EASTUS2EUAP:20260330T065415Z:cf5d91a3-a913-4d5b-9164-bccf81eb8eee", ], "X-Content-Type-Options": ["nosniff"], "Date": [ "Mon, 30 Mar 2026 06:54:14 GMT", ], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Expires": ["-1"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Msedge-Ref": [ "Ref A: FED97ACCAC0E49C1AB5718A871C140C7 Ref B: CHI261140514023 Ref C: 2026-03-30T06:54:14Z", ], "Content-Length": ["655"], "Content-Type": [ "application/json; charset=utf-8", ], "X-Cache": ["CONFIG_NOCACHE"], }, Body: <exported.nopClosingBytesReader \| 0xc0012c8040>{ s: "{\"error\":{\"code\":\"InvalidTemplateDeployment\",\"message\":\"The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.\",\"details\":[{\"code\":\"ResourceCountExceedsLimitDueToTemplate\",\"message\":\"Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress\",\"detai... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 06:09:05 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:100]: Unexpected error: <fmt.wrapError \| 0xc00068d580>: failed to create HCP cluster cluster-zstream-4-21-wggw2c: failed waiting for cluster="cluster-zstream-4-21-wggw2c" in resourcegroup="rg-zstream-upgrade-4-21-6hp9ww" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029", "name": "d05a5529-ae05-46fe-83bf-87a75636e029", "status": "Failed", "startTime": "2026-03-30T06:26:33.107623699Z", "endTime": "2026-03-30T06:27:03.896795282Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-wggw2c: failed waiting for cluster=\"cluster-zstream-4-21-wggw2c\" in resourcegroup=\"rg-zstream-upgrade-4-21-6hp9ww\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029\",\n \"name\": \"d05a5529-ae05-46fe-83bf-87a75636e029\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:26:33.107623699Z\",\n \"endTime\": \"2026-03-30T06:27:03.896795282Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00068d560>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-wggw2c\" in resourcegroup=\"rg-zstream-upgrade-4-21-6hp9ww\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029\",\n \"name\": \"d05a5529-ae05-46fe-83bf-87a75636e029\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:26:33.107623699Z\",\n \"endTime\": \"2026-03-30T06:27:03.896795282Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012c8bd0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "4e624f15-373d-4b5d-9aa2-5728bfa533ba", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/ddf1cb1e-19cd-44b1-a6fc-7ee87af76e16", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"], "X-Ms-Correlation-Request-Id": [ "9f65d160-79d1-46b3-8b9c-23b37e9e9527", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: E48A2BDBE5144E9E9D51043FBB540576 Ref B: CHI261140513040 Ref C: 2026-03-30T06:27:04Z", ], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:100]: Unexpected error: <fmt.wrapError \| 0xc00068d580>: failed to create HCP cluster cluster-zstream-4-21-wggw2c: failed waiting for cluster="cluster-zstream-4-21-wggw2c" in resourcegroup="rg-zstream-upgrade-4-21-6hp9ww" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029", "name": "d05a5529-ae05-46fe-83bf-87a75636e029", "status": "Failed", "startTime": "2026-03-30T06:26:33.107623699Z", "endTime": "2026-03-30T06:27:03.896795282Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-wggw2c: failed waiting for cluster=\"cluster-zstream-4-21-wggw2c\" in resourcegroup=\"rg-zstream-upgrade-4-21-6hp9ww\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029\",\n \"name\": \"d05a5529-ae05-46fe-83bf-87a75636e029\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:26:33.107623699Z\",\n \"endTime\": \"2026-03-30T06:27:03.896795282Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00068d560>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-wggw2c\" in resourcegroup=\"rg-zstream-upgrade-4-21-6hp9ww\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029\",\n \"name\": \"d05a5529-ae05-46fe-83bf-87a75636e029\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:26:33.107623699Z\",\n \"endTime\": \"2026-03-30T06:27:03.896795282Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012c8bd0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "4e624f15-373d-4b5d-9aa2-5728bfa533ba", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/ddf1cb1e-19cd-44b1-a6fc-7ee87af76e16", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"], "X-Ms-Correlation-Request-Id": [ "9f65d160-79d1-46b3-8b9c-23b37e9e9527", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: E48A2BDBE5144E9E9D51043FBB540576 Ref B: CHI261140513040 Ref C: 2026-03-30T06:27:04Z", ], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 06:09:05 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc0011eebe0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-4rvwkj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91", "name": "db0f712c-aa43-42f2-ba74-2b516fee6d91", "status": "Failed", "startTime": "2026-03-30T06:20:12.516389794Z", "endTime": "2026-03-30T06:20:48.848822062Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4rvwkj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91\",\n \"name\": \"db0f712c-aa43-42f2-ba74-2b516fee6d91\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:20:12.516389794Z\",\n \"endTime\": \"2026-03-30T06:20:48.848822062Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0011eebc0>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4rvwkj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91\",\n \"name\": \"db0f712c-aa43-42f2-ba74-2b516fee6d91\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:20:12.516389794Z\",\n \"endTime\": \"2026-03-30T06:20:48.848822062Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0015993e0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "783c170b-8f66-4831-be6b-bc51973e70b7", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/d9f54903-b7b1-4721-9c43-2cc2e56ffb60", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"], "Pragma": ["no-cache"], "Content-Length": ["503"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"], "Cache-Control": ["no-cache"], "Expires": ["-1"], "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Correlation-Request-Id": [ "44d4e8c9-3387-4e0a-b086-ecf8fbc2f225", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T062053Z:44d4e8c9-3387-4e0a-b086-ecf8fbc2f225", ], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc0011eebe0>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-4rvwkj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91", "name": "db0f712c-aa43-42f2-ba74-2b516fee6d91", "status": "Failed", "startTime": "2026-03-30T06:20:12.516389794Z", "endTime": "2026-03-30T06:20:48.848822062Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4rvwkj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91\",\n \"name\": \"db0f712c-aa43-42f2-ba74-2b516fee6d91\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:20:12.516389794Z\",\n \"endTime\": \"2026-03-30T06:20:48.848822062Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0011eebc0>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4rvwkj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91\",\n \"name\": \"db0f712c-aa43-42f2-ba74-2b516fee6d91\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:20:12.516389794Z\",\n \"endTime\": \"2026-03-30T06:20:48.848822062Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0015993e0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "783c170b-8f66-4831-be6b-bc51973e70b7", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/d9f54903-b7b1-4721-9c43-2cc2e56ffb60", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"], "Pragma": ["no-cache"], "Content-Length": ["503"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"], "Cache-Control": ["no-cache"], "Expires": ["-1"], "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Correlation-Request-Id": [ "44d4e8c9-3387-4e0a-b086-ecf8fbc2f225", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T062053Z:44d4e8c9-3387-4e0a-b086-ecf8fbc2f225", ], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 06:09:05 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapError \| 0xc000c0fd00>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-lsx2ks" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279", "name": "0f618886-6ee0-48de-9547-a72389e82279", "status": "Failed", "startTime": "2026-03-30T06:18:28.258152765Z", "endTime": "2026-03-30T06:18:44.003768858Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lsx2ks\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279\",\n \"name\": \"0f618886-6ee0-48de-9547-a72389e82279\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:18:28.258152765Z\",\n \"endTime\": \"2026-03-30T06:18:44.003768858Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000c0fc60>{ msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lsx2ks\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279\",\n \"name\": \"0f618886-6ee0-48de-9547-a72389e82279\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:18:28.258152765Z\",\n \"endTime\": \"2026-03-30T06:18:44.003768858Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00106a720>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Content-Type-Options": ["nosniff"], "Date": [ "Mon, 30 Mar 2026 06:18:48 GMT", ], "Content-Type": ["application/json"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Correlation-Request-Id": [ "022cb22e-6859-4c7f-a168-ebde742732c9", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Cache-Control": ["no-cache"], "Content-Length": ["503"], "Expires": ["-1"], "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T061849Z:022cb22e-6859-4c7f-a168-ebde742732c9", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapError \| 0xc000c0fd00>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-lsx2ks" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279", "name": "0f618886-6ee0-48de-9547-a72389e82279", "status": "Failed", "startTime": "2026-03-30T06:18:28.258152765Z", "endTime": "2026-03-30T06:18:44.003768858Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lsx2ks\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279\",\n \"name\": \"0f618886-6ee0-48de-9547-a72389e82279\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:18:28.258152765Z\",\n \"endTime\": \"2026-03-30T06:18:44.003768858Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000c0fc60>{ msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lsx2ks\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279\",\n \"name\": \"0f618886-6ee0-48de-9547-a72389e82279\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:18:28.258152765Z\",\n \"endTime\": \"2026-03-30T06:18:44.003768858Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00106a720>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Content-Type-Options": ["nosniff"], "Date": [ "Mon, 30 Mar 2026 06:18:48 GMT", ], "Content-Type": ["application/json"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Correlation-Request-Id": [ "022cb22e-6859-4c7f-a168-ebde742732c9", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Cache-Control": ["no-cache"], "Content-Length": ["503"], "Expires": ["-1"], "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260330T061849Z:022cb22e-6859-4c7f-a168-ebde742732c9", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 06:09:05 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error: <fmt.wrapErrors \| 0xc00160a6f0>: failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-zd6bdf" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: context deadline exceeded { msg: "failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-zd6bdf\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0015092b0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf", }, <fmt.wrapErrors \| 0xc00160a6c0>{ msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-zd6bdf\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0015092b0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error: <fmt.wrapErrors \| 0xc00160a6f0>: failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-zd6bdf" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: context deadline exceeded { msg: "failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-zd6bdf\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0015092b0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf", }, <fmt.wrapErrors \| 0xc00160a6c0>{ msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-zd6bdf\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0015092b0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf", }, <context.deadlineExceededError>{}, ], }, ], } occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 06:09:05 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: Unexpected error: <fmt.wrapError \| 0xc0016131a0>: failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-8qw9d2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e", "name": "e8c3c061-19f1-4d72-b474-3fbe98891e9e", "status": "Failed", "startTime": "2026-03-30T06:18:33.483509413Z", "endTime": "2026-03-30T06:18:44.018298845Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8qw9d2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n \"name\": \"e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:18:33.483509413Z\",\n \"endTime\": \"2026-03-30T06:18:44.018298845Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001613180>{ msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8qw9d2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n \"name\": \"e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:18:33.483509413Z\",\n \"endTime\": \"2026-03-30T06:18:44.018298845Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00157d260>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Routing-Request-Id": [ "CANADAEAST:20260330T061854Z:72e130e2-01fe-4e82-86ac-3810c7b0b18a", ], "Date": [ "Mon, 30 Mar 2026 06:18:53 GMT", ], "X-Ms-Correlation-Request-Id": [ "72e130e2-01fe-4e82-86ac-3810c7b0b18a", ], "Pragma": ["no-cache"], "Content-Type": ["application/json"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: 05A3F815056F4198B1B5D4420C306AED Ref B: CHI261140504023 Ref C: 2026-03-30T06:18:53Z", ], "Cache-Control": ["no-cache"], "X-Ms-Request-Id": [ "111c62... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: Unexpected error: <fmt.wrapError \| 0xc0016131a0>: failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-8qw9d2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e", "name": "e8c3c061-19f1-4d72-b474-3fbe98891e9e", "status": "Failed", "startTime": "2026-03-30T06:18:33.483509413Z", "endTime": "2026-03-30T06:18:44.018298845Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8qw9d2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n \"name\": \"e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:18:33.483509413Z\",\n \"endTime\": \"2026-03-30T06:18:44.018298845Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001613180>{ msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8qw9d2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n \"name\": \"e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-30T06:18:33.483509413Z\",\n \"endTime\": \"2026-03-30T06:18:44.018298845Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00157d260>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Routing-Request-Id": [ "CANADAEAST:20260330T061854Z:72e130e2-01fe-4e82-86ac-3810c7b0b18a", ], "Date": [ "Mon, 30 Mar 2026 06:18:53 GMT", ], "X-Ms-Correlation-Request-Id": [ "72e130e2-01fe-4e82-86ac-3810c7b0b18a", ], "Pragma": ["no-cache"], "Content-Type": ["application/json"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: 05A3F815056F4198B1B5D4420C306AED Ref B: CHI261140504023 Ref C: 2026-03-30T06:18:53Z", ], "Cache-Control": ["no-cache"], "X-Ms-Request-Id": [ "111c62... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
05:13:13	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
05:04:47	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
04:55:35	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
02:32:50	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
01:36:12	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
01:32:30	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
00:51:33	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
00:47:45	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	1	e2e (1) timeout during CreateHCPClusterAndWait; context deadline exceeded 00:47:45 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error: <fmt.wrapErrors \| 0xc000f732f0>: failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-8k227b, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-8k227b" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-8k227b, error: context deadline exceeded { msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-8k227b, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-8k227b\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-8k227b, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0010a7a30>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-8k227b", }, <fmt.wrapErrors \| 0xc000f732c0>{ msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-8k227b\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-8k227b, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0010a7a30>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-8k227b", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error: <fmt.wrapErrors \| 0xc000f732f0>: failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-8k227b, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-8k227b" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-8k227b, error: context deadline exceeded { msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-8k227b, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-8k227b\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-8k227b, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0010a7a30>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-8k227b", }, <fmt.wrapErrors \| 0xc000f732c0>{ msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-8k227b\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-8k227b, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0010a7a30>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-8k227b", }, <context.deadlineExceededError>{}, ], }, ], } occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

06:09:05 UTC · failed at e2e · test Create HCPOpenShiftCluster with Private KeyVault should create a cluster with private keyvault using v20251223preview API · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:135]: Unexpected error:
    <*fmt.wrapErrors | 0xc001606750>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-zrnjn5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-zrnjn5, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-zrnjn5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-zrnjn5, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000821bb0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-zrnjn5",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_private_kv.go:135]: Unexpected error:
    <*fmt.wrapErrors | 0xc001606750>: 
    failed waiting for cluster="private-kv-cluster" in resourcegroup="private-keyvault-zrnjn5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-zrnjn5, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"private-kv-cluster\" in resourcegroup=\"private-keyvault-zrnjn5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-zrnjn5, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000821bb0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPCluster20251223AndWait for cluster private-kv-cluster in resource group private-keyvault-zrnjn5",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

06:09:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ee85d0>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-2qwhkh" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-2qwhkh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011cf9e0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh",
            },
            <*fmt.wrapErrors | 0xc000ee85a0>{
                msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-2qwhkh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0011cf9e0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ee85d0>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-2qwhkh" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-2qwhkh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011cf9e0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh",
            },
            <*fmt.wrapErrors | 0xc000ee85a0>{
                msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-2qwhkh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0011cf9e0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-2qwhkh",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

06:09:05 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:117]: Unexpected error:
    <*fmt.wrapError | 0xc000e16f40>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-5p4qlp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a",
      "name": "cfc5391b-7b48-4657-9b50-67b79747ab7a",
      "status": "Failed",
      "startTime": "2026-03-30T06:22:25.344609849Z",
      "endTime": "2026-03-30T06:22:55.779876405Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5p4qlp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a\",\n  \"name\": \"cfc5391b-7b48-4657-9b50-67b79747ab7a\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:22:25.344609849Z\",\n  \"endTime\": \"2026-03-30T06:22:55.779876405Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000e16f20>{
            msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5p4qlp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a\",\n  \"name\": \"cfc5391b-7b48-4657-9b50-67b79747ab7a\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:22:25.344609849Z\",\n  \"endTime\": \"2026-03-30T06:22:55.779876405Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000fe3860>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["503"],
                        "X-Ms-Request-Id": [
                            "141c0618-2615-4e89-98c8-24cc9f4c8553",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Correlation-Request-Id": [
                            "f073cfd3-865e-46e9-9354-23fc2d617e27",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T062256Z:f073cfd3-865e-46e9-9354-23fc2d617e27",
                        ],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Msedge-Ref": [
                            "Ref A: CDC0382112E64ECDBEECE2C1D1A01FE0 Ref B: CH1AA2020617025 Ref C: 2026-03-30T06:22:56Z",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Content-Type-Options": ["nosniff"],
                ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:117]: Unexpected error:
    <*fmt.wrapError | 0xc000e16f40>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-5p4qlp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a",
      "name": "cfc5391b-7b48-4657-9b50-67b79747ab7a",
      "status": "Failed",
      "startTime": "2026-03-30T06:22:25.344609849Z",
      "endTime": "2026-03-30T06:22:55.779876405Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5p4qlp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a\",\n  \"name\": \"cfc5391b-7b48-4657-9b50-67b79747ab7a\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:22:25.344609849Z\",\n  \"endTime\": \"2026-03-30T06:22:55.779876405Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000e16f20>{
            msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-5p4qlp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/cfc5391b-7b48-4657-9b50-67b79747ab7a\",\n  \"name\": \"cfc5391b-7b48-4657-9b50-67b79747ab7a\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:22:25.344609849Z\",\n  \"endTime\": \"2026-03-30T06:22:55.779876405Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000fe3860>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["503"],
                        "X-Ms-Request-Id": [
                            "141c0618-2615-4e89-98c8-24cc9f4c8553",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Correlation-Request-Id": [
                            "f073cfd3-865e-46e9-9354-23fc2d617e27",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T062256Z:f073cfd3-865e-46e9-9354-23fc2d617e27",
                        ],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Msedge-Ref": [
                            "Ref A: CDC0382112E64ECDBEECE2C1D1A01FE0 Ref B: CH1AA2020617025 Ref C: 2026-03-30T06:22:56Z",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Content-Type-Options": ["nosniff"],
                ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

06:09:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error:
    <*fmt.wrapErrors | 0xc00148d140>: 
    failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-mww9kq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-mww9kq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0012346f0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq",
            },
            <*fmt.wrapErrors | 0xc00148d0e0>{
                msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-mww9kq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0012346f0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error:
    <*fmt.wrapErrors | 0xc00148d140>: 
    failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-mww9kq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-mww9kq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0012346f0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq",
            },
            <*fmt.wrapErrors | 0xc00148d0e0>{
                msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-mww9kq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0012346f0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-mww9kq",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

06:09:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage ImageDigestMirrors · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
    <*fmt.wrapError | 0xc000c50ee0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-n5nffq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b",
      "name": "1517a95b-a81b-4ce8-841d-22153214295b",
      "status": "Failed",
      "startTime": "2026-03-30T06:15:55.940417137Z",
      "endTime": "2026-03-30T06:16:38.146603108Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n5nffq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b\",\n  \"name\": \"1517a95b-a81b-4ce8-841d-22153214295b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:15:55.940417137Z\",\n  \"endTime\": \"2026-03-30T06:16:38.146603108Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000c50ec0>{
            msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n5nffq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b\",\n  \"name\": \"1517a95b-a81b-4ce8-841d-22153214295b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:15:55.940417137Z\",\n  \"endTime\": \"2026-03-30T06:16:38.146603108Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0004c5650>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["503"],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Content-Type-Options": ["nosniff"],
                        "Date": [
                            "Mon, 30 Mar 2026 06:16:46 GMT",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/28c44e1a-d416-41d6-a0fc-92aed87244cd",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "ec54b0dc-0811-4ecd-92b7-fd2310698c78",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T061647Z:ec54b0dc-0811-4ecd-92b7-fd2310698c78",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
               ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/idms_lifecycle.go:136]: Unexpected error:
    <*fmt.wrapError | 0xc000c50ee0>: 
    failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster="idms-e2e-hcp-cluster" in resourcegroup="idms-n5nffq" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b",
      "name": "1517a95b-a81b-4ce8-841d-22153214295b",
      "status": "Failed",
      "startTime": "2026-03-30T06:15:55.940417137Z",
      "endTime": "2026-03-30T06:16:38.146603108Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster idms-e2e-hcp-cluster: failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n5nffq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b\",\n  \"name\": \"1517a95b-a81b-4ce8-841d-22153214295b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:15:55.940417137Z\",\n  \"endTime\": \"2026-03-30T06:16:38.146603108Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000c50ec0>{
            msg: "failed waiting for cluster=\"idms-e2e-hcp-cluster\" in resourcegroup=\"idms-n5nffq\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1517a95b-a81b-4ce8-841d-22153214295b\",\n  \"name\": \"1517a95b-a81b-4ce8-841d-22153214295b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:15:55.940417137Z\",\n  \"endTime\": \"2026-03-30T06:16:38.146603108Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0004c5650>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["503"],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Content-Type-Options": ["nosniff"],
                        "Date": [
                            "Mon, 30 Mar 2026 06:16:46 GMT",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/28c44e1a-d416-41d6-a0fc-92aed87244cd",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "ec54b0dc-0811-4ecd-92b7-fd2310698c78",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T061647Z:ec54b0dc-0811-4ecd-92b7-fd2310698c78",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
               ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

06:09:05 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc00165cd20>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-7cfr2p" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-7cfr2p\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e18a90>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p",
            },
            <*fmt.wrapErrors | 0xc00165ccf0>{
                msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-7cfr2p\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e18a90>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc00165cd20>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-7cfr2p" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-7cfr2p\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e18a90>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p",
            },
            <*fmt.wrapErrors | 0xc00165ccf0>{
                msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-7cfr2p\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e18a90>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-7cfr2p",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

06:09:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc001198d50>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-rqfwmq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rqfwmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00111ad80>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq",
            },
            <*fmt.wrapErrors | 0xc001198d20>{
                msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rqfwmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00111ad80>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc001198d50>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-rqfwmq" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rqfwmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00111ad80>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq",
            },
            <*fmt.wrapErrors | 0xc001198d20>{
                msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-rqfwmq\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00111ad80>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-rqfwmq",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

06:09:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapError | 0xc000e0b7c0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-l2s4f2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02",
      "name": "f9df063e-f124-4f77-8a25-358c9d5cde02",
      "status": "Failed",
      "startTime": "2026-03-30T06:16:32.010035382Z",
      "endTime": "2026-03-30T06:16:48.681254232Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-l2s4f2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\",\n  \"name\": \"f9df063e-f124-4f77-8a25-358c9d5cde02\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:16:32.010035382Z\",\n  \"endTime\": \"2026-03-30T06:16:48.681254232Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000e0b7a0>{
            msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-l2s4f2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\",\n  \"name\": \"f9df063e-f124-4f77-8a25-358c9d5cde02\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:16:32.010035382Z\",\n  \"endTime\": \"2026-03-30T06:16:48.681254232Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00104d650>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T061652Z:5fa8b3d2-fffd-493a-9339-7cb51375d257",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "Date": [
                            "Mon, 30 Mar 2026 06:16:51 GMT",
                        ],
                        "Expires": ["-1"],
                        "X-Msedge-Ref": [
                            "Ref A: 9C0B547CA14B41D4A16FDC03FE815989 Ref B: CHI261140514036 Ref C: 2026-03-30T06:16:52Z",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
      ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapError | 0xc000e0b7c0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-l2s4f2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02",
      "name": "f9df063e-f124-4f77-8a25-358c9d5cde02",
      "status": "Failed",
      "startTime": "2026-03-30T06:16:32.010035382Z",
      "endTime": "2026-03-30T06:16:48.681254232Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-l2s4f2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\",\n  \"name\": \"f9df063e-f124-4f77-8a25-358c9d5cde02\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:16:32.010035382Z\",\n  \"endTime\": \"2026-03-30T06:16:48.681254232Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000e0b7a0>{
            msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-l2s4f2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f9df063e-f124-4f77-8a25-358c9d5cde02\",\n  \"name\": \"f9df063e-f124-4f77-8a25-358c9d5cde02\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:16:32.010035382Z\",\n  \"endTime\": \"2026-03-30T06:16:48.681254232Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00104d650>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T061652Z:5fa8b3d2-fffd-493a-9339-7cb51375d257",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "Date": [
                            "Mon, 30 Mar 2026 06:16:51 GMT",
                        ],
                        "Expires": ["-1"],
                        "X-Msedge-Ref": [
                            "Ref A: 9C0B547CA14B41D4A16FDC03FE815989 Ref B: CHI261140514036 Ref C: 2026-03-30T06:16:52Z",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
      ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

06:09:05 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error:
    <*fmt.wrapError | 0xc00082f8a0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-sx6f6k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4",
      "name": "75078195-e41b-452a-877c-5d9b8e76e3e4",
      "status": "Failed",
      "startTime": "2026-03-30T06:20:53.192418254Z",
      "endTime": "2026-03-30T06:21:52.368868636Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-sx6f6k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4\",\n  \"name\": \"75078195-e41b-452a-877c-5d9b8e76e3e4\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:20:53.192418254Z\",\n  \"endTime\": \"2026-03-30T06:21:52.368868636Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00082f880>{
            msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-sx6f6k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4\",\n  \"name\": \"75078195-e41b-452a-877c-5d9b8e76e3e4\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:20:53.192418254Z\",\n  \"endTime\": \"2026-03-30T06:21:52.368868636Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0014c9ec0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Cache-Control": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "6a949a5b-9896-4995-9b24-95e48c1486b0",
                        ],
                        "X-Envoy-Upstream-Service-Time": ["19"],
                        "Pragma": ["no-cache"],
                        "Content-Length": ["503"],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T062154Z:51754cc2-4bfb-40ab-9d75-1f562f630654",
                        ],
                        "Date": [
                            "Mon, 30 Mar 2026 06:21:54 GMT",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=c94dda6b-80b4-4144-a102-590...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error:
    <*fmt.wrapError | 0xc00082f8a0>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-sx6f6k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4",
      "name": "75078195-e41b-452a-877c-5d9b8e76e3e4",
      "status": "Failed",
      "startTime": "2026-03-30T06:20:53.192418254Z",
      "endTime": "2026-03-30T06:21:52.368868636Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-sx6f6k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4\",\n  \"name\": \"75078195-e41b-452a-877c-5d9b8e76e3e4\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:20:53.192418254Z\",\n  \"endTime\": \"2026-03-30T06:21:52.368868636Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00082f880>{
            msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-sx6f6k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/75078195-e41b-452a-877c-5d9b8e76e3e4\",\n  \"name\": \"75078195-e41b-452a-877c-5d9b8e76e3e4\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:20:53.192418254Z\",\n  \"endTime\": \"2026-03-30T06:21:52.368868636Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0014c9ec0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Cache-Control": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "6a949a5b-9896-4995-9b24-95e48c1486b0",
                        ],
                        "X-Envoy-Upstream-Service-Time": ["19"],
                        "Pragma": ["no-cache"],
                        "Content-Length": ["503"],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T062154Z:51754cc2-4bfb-40ab-9d75-1f562f630654",
                        ],
                        "Date": [
                            "Mon, 30 Mar 2026 06:21:54 GMT",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=c94dda6b-80b4-4144-a102-590...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Cluster provisioning failed

06:09:05 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:153]: Cluster provisioning failed
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:153]: Cluster provisioning failed

timeout during CreateHCPClusterAndWait; context deadline exceeded

06:09:05 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
    <*fmt.wrapErrors | 0xc001454c30>: 
    failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-528g9r" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-528g9r\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e7cad0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r",
            },
            <*fmt.wrapErrors | 0xc001454c00>{
                msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-528g9r\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e7cad0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
    <*fmt.wrapErrors | 0xc001454c30>: 
    failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-528g9r" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster patch-tags-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-528g9r\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000e7cad0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r",
            },
            <*fmt.wrapErrors | 0xc001454c00>{
                msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-528g9r\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000e7cad0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-tags-cluster in resource group patch-tags-528g9r",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

06:09:05 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapError | 0xc00099c3e0>: 
    failed to create HCP cluster ea-listdjh5pl: failed waiting for cluster="ea-listdjh5pl" in resourcegroup="ea-list-vc9fmh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41",
      "name": "d92e5056-78d6-404e-a6ae-95d01a203d41",
      "status": "Failed",
      "startTime": "2026-03-30T06:28:34.388150903Z",
      "endTime": "2026-03-30T06:29:01.410266778Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ea-listdjh5pl: failed waiting for cluster=\"ea-listdjh5pl\" in resourcegroup=\"ea-list-vc9fmh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41\",\n  \"name\": \"d92e5056-78d6-404e-a6ae-95d01a203d41\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:28:34.388150903Z\",\n  \"endTime\": \"2026-03-30T06:29:01.410266778Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00099c3c0>{
            msg: "failed waiting for cluster=\"ea-listdjh5pl\" in resourcegroup=\"ea-list-vc9fmh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41\",\n  \"name\": \"d92e5056-78d6-404e-a6ae-95d01a203d41\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:28:34.388150903Z\",\n  \"endTime\": \"2026-03-30T06:29:01.410266778Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00128a3f0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Correlation-Request-Id": [
                            "10073257-81d6-4a5d-9336-e607cce74141",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: 93FDCB2B04AE464DBCEFBE7C2F6542A0 Ref B: CHI261140502034 Ref C: 2026-03-30T06:29:05Z",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "3d7e90c1-6210-4595-a613-3eb6e9018232",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96db...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapError | 0xc00099c3e0>: 
    failed to create HCP cluster ea-listdjh5pl: failed waiting for cluster="ea-listdjh5pl" in resourcegroup="ea-list-vc9fmh" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41",
      "name": "d92e5056-78d6-404e-a6ae-95d01a203d41",
      "status": "Failed",
      "startTime": "2026-03-30T06:28:34.388150903Z",
      "endTime": "2026-03-30T06:29:01.410266778Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ea-listdjh5pl: failed waiting for cluster=\"ea-listdjh5pl\" in resourcegroup=\"ea-list-vc9fmh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41\",\n  \"name\": \"d92e5056-78d6-404e-a6ae-95d01a203d41\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:28:34.388150903Z\",\n  \"endTime\": \"2026-03-30T06:29:01.410266778Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00099c3c0>{
            msg: "failed waiting for cluster=\"ea-listdjh5pl\" in resourcegroup=\"ea-list-vc9fmh\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d92e5056-78d6-404e-a6ae-95d01a203d41\",\n  \"name\": \"d92e5056-78d6-404e-a6ae-95d01a203d41\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:28:34.388150903Z\",\n  \"endTime\": \"2026-03-30T06:29:01.410266778Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00128a3f0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Correlation-Request-Id": [
                            "10073257-81d6-4a5d-9336-e607cce74141",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: 93FDCB2B04AE464DBCEFBE7C2F6542A0 Ref B: CHI261140502034 Ref C: 2026-03-30T06:29:05Z",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "3d7e90c1-6210-4595-a613-3eb6e9018232",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96db...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

06:09:05 UTC · failed at e2e · test Nodepool Ephemeral OS Disk should create a nodepool with ephemeral OS disk when autoRepair is enabled · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error:
    <*fmt.wrapError | 0xc00089a340>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-gsvbfn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8",
      "name": "44da3a3a-6183-4e59-9edc-5597fe1650b8",
      "status": "Failed",
      "startTime": "2026-03-30T06:20:01.753593452Z",
      "endTime": "2026-03-30T06:20:48.88837796Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ephemeral-disk: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-gsvbfn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8\",\n  \"name\": \"44da3a3a-6183-4e59-9edc-5597fe1650b8\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:20:01.753593452Z\",\n  \"endTime\": \"2026-03-30T06:20:48.88837796Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00089a320>{
            msg: "failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-gsvbfn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8\",\n  \"name\": \"44da3a3a-6183-4e59-9edc-5597fe1650b8\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:20:01.753593452Z\",\n  \"endTime\": \"2026-03-30T06:20:48.88837796Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0008641b0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["502"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"],
                        "Date": [
                            "Mon, 30 Mar 2026 06:20:52 GMT",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "a3b37884-de6d-4b7e-bb88-9da4257ef455",
                        ],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Correlation-Request-Id": [
                            "eb37ef8b-2459-4a5c-9a4e-c2ad42298d82",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T062052Z:eb37ef8b-2459-4a5c-9a4e-c2ad42298d82",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Msedge-Ref": [
                            "Ref A: 17B0B8CD7B0E4D9BB4BC8B91EFCA7646 Ref B: CH1AA2020617019 Ref C: 2026-03-30T06:20:52Z...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_ephemeral_osdisk.go:112]: Unexpected error:
    <*fmt.wrapError | 0xc00089a340>: 
    failed to create HCP cluster ephemeral-disk: failed waiting for cluster="ephemeral-disk" in resourcegroup="ephemeral-osdisk-gsvbfn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8",
      "name": "44da3a3a-6183-4e59-9edc-5597fe1650b8",
      "status": "Failed",
      "startTime": "2026-03-30T06:20:01.753593452Z",
      "endTime": "2026-03-30T06:20:48.88837796Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ephemeral-disk: failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-gsvbfn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8\",\n  \"name\": \"44da3a3a-6183-4e59-9edc-5597fe1650b8\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:20:01.753593452Z\",\n  \"endTime\": \"2026-03-30T06:20:48.88837796Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00089a320>{
            msg: "failed waiting for cluster=\"ephemeral-disk\" in resourcegroup=\"ephemeral-osdisk-gsvbfn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/44da3a3a-6183-4e59-9edc-5597fe1650b8\",\n  \"name\": \"44da3a3a-6183-4e59-9edc-5597fe1650b8\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:20:01.753593452Z\",\n  \"endTime\": \"2026-03-30T06:20:48.88837796Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0008641b0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["502"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"],
                        "Date": [
                            "Mon, 30 Mar 2026 06:20:52 GMT",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "a3b37884-de6d-4b7e-bb88-9da4257ef455",
                        ],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Correlation-Request-Id": [
                            "eb37ef8b-2459-4a5c-9a4e-c2ad42298d82",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T062052Z:eb37ef8b-2459-4a5c-9a4e-c2ad42298d82",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Msedge-Ref": [
                            "Ref A: 17B0B8CD7B0E4D9BB4BC8B91EFCA7646 Ref B: CH1AA2020617019 Ref C: 2026-03-30T06:20:52Z...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: InternalServerError; detail code AuthorizationFailed; detail message The client '<uuid>' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network

06:09:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapError | 0xc000334400>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-57zxqg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31",
      "name": "3c3b1341-fda3-41e9-8dd4-5c30342a2d31",
      "status": "Failed",
      "startTime": "2026-03-30T06:12:39.328826001Z",
      "endTime": "2026-03-30T06:13:08.581005823Z",
      "error": {
        "code": "InternalServerError",
        "message": "Failed to get virtual network 'customer-vnet-name-4-19' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client '3820d679-b809-47f8-8f80-79cbd25d1d89' with object id '62b94c2a-37fd-4b0a-b97c-1370b69f6c0c' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-57zxqg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31\",\n  \"name\": \"3c3b1341-fda3-41e9-8dd4-5c30342a2d31\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:12:39.328826001Z\",\n  \"endTime\": \"2026-03-30T06:13:08.581005823Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-name-4-19' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client '3820d679-b809-47f8-8f80-79cbd25d1d89' with object id '62b94c2a-37fd-4b0a-b97c-1370b69f6c0c' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc001475530>{
            ErrorCode: "InternalServerError",
            StatusCode: 200,
            RawResponse: {
                Status: "200 OK",
                StatusCode: 200,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Pragma": ["no-cache"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/f58e194a-3f86-4008-98dc-aa5f60802195",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Msedge-Ref": [
                        "Ref A: 60DFA7F6143244D1924498FA54547EC7 Ref B: CHI261140510034 Ref C: 2026-03-30T06:13:10Z",
                    ],
                    "Date": [
                        "Mon, 30 Mar 2026 06:13:10 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Expires": ["-1"],
                    "X-Ms-Correlation-Request-Id": [
                        "bdec501d-0ca0-4486-8500-77afaa68e37a",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "CANADAEAST:20260330T061310Z:bdec501d-0ca0-4486-8500-77afaa68e37a",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["9"],
                    "Content-Length": ["1563"],
                    "Content-Type": ["application/json"],
                    "X-M...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapError | 0xc000334400>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-57zxqg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31",
      "name": "3c3b1341-fda3-41e9-8dd4-5c30342a2d31",
      "status": "Failed",
      "startTime": "2026-03-30T06:12:39.328826001Z",
      "endTime": "2026-03-30T06:13:08.581005823Z",
      "error": {
        "code": "InternalServerError",
        "message": "Failed to get virtual network 'customer-vnet-name-4-19' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client '3820d679-b809-47f8-8f80-79cbd25d1d89' with object id '62b94c2a-37fd-4b0a-b97c-1370b69f6c0c' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-57zxqg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/3c3b1341-fda3-41e9-8dd4-5c30342a2d31\",\n  \"name\": \"3c3b1341-fda3-41e9-8dd4-5c30342a2d31\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:12:39.328826001Z\",\n  \"endTime\": \"2026-03-30T06:13:08.581005823Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"Failed to get virtual network 'customer-vnet-name-4-19' details: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19\\n--------------------------------------------------------------------------------\\nRESPONSE 403: 403 Forbidden\\nERROR CODE: AuthorizationFailed\\n--------------------------------------------------------------------------------\\n{\\n  \\\"error\\\": {\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\n    \\\"message\\\": \\\"The client '3820d679-b809-47f8-8f80-79cbd25d1d89' with object id '62b94c2a-37fd-4b0a-b97c-1370b69f6c0c' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/rg-cluster-back-version-57zxqg/providers/Microsoft.Network/virtualNetworks/customer-vnet-name-4-19' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\n  }\\n}\\n--------------------------------------------------------------------------------\\n\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc001475530>{
            ErrorCode: "InternalServerError",
            StatusCode: 200,
            RawResponse: {
                Status: "200 OK",
                StatusCode: 200,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "Pragma": ["no-cache"],
                    "X-Ms-Operation-Identifier": [
                        "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/f58e194a-3f86-4008-98dc-aa5f60802195",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "X-Msedge-Ref": [
                        "Ref A: 60DFA7F6143244D1924498FA54547EC7 Ref B: CHI261140510034 Ref C: 2026-03-30T06:13:10Z",
                    ],
                    "Date": [
                        "Mon, 30 Mar 2026 06:13:10 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Expires": ["-1"],
                    "X-Ms-Correlation-Request-Id": [
                        "bdec501d-0ca0-4486-8500-77afaa68e37a",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "CANADAEAST:20260330T061310Z:bdec501d-0ca0-4486-8500-77afaa68e37a",
                    ],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Envoy-Upstream-Service-Time": ["9"],
                    "Content-Length": ["1563"],
                    "Content-Type": ["application/json"],
                    "X-M...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

06:09:05 UTC · failed at e2e · test Customer should be able to use workload identity via the cluster OIDC issuer URL · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: Unexpected error:
    <*fmt.wrapError | 0xc000a0b040>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-t4849k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395",
      "name": "671489be-ab4c-40e2-81b0-625f59cf6395",
      "status": "Failed",
      "startTime": "2026-03-30T06:22:24.890658519Z",
      "endTime": "2026-03-30T06:22:55.769894295Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-t4849k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\",\n  \"name\": \"671489be-ab4c-40e2-81b0-625f59cf6395\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:22:24.890658519Z\",\n  \"endTime\": \"2026-03-30T06:22:55.769894295Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a0b020>{
            msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-t4849k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\",\n  \"name\": \"671489be-ab4c-40e2-81b0-625f59cf6395\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:22:24.890658519Z\",\n  \"endTime\": \"2026-03-30T06:22:55.769894295Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0011f0ed0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/be562cea-8623-4678-adaa-0057ff53bbd9",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "8b2df379-7ba3-4320-8e92-21a566e40b6c",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: BC420E598D454882AFE3F4518AA17A00 Ref B: CHI261140512052 Ref C: 2026-03-30T06:22:55Z",
                        ],
                        "Content-Length": ["503"],
                        "X-Ms-Request-Id": [
                            "fef2e839-309a-4189-8dfd-f18f57fe8ccf",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Rema...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/oidc_issuer_workload_identity.go:216]: Unexpected error:
    <*fmt.wrapError | 0xc000a0b040>: 
    failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster="oidc-wi-cluster" in resourcegroup="oidc-wi-t4849k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395",
      "name": "671489be-ab4c-40e2-81b0-625f59cf6395",
      "status": "Failed",
      "startTime": "2026-03-30T06:22:24.890658519Z",
      "endTime": "2026-03-30T06:22:55.769894295Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster oidc-wi-cluster: failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-t4849k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\",\n  \"name\": \"671489be-ab4c-40e2-81b0-625f59cf6395\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:22:24.890658519Z\",\n  \"endTime\": \"2026-03-30T06:22:55.769894295Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a0b020>{
            msg: "failed waiting for cluster=\"oidc-wi-cluster\" in resourcegroup=\"oidc-wi-t4849k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/671489be-ab4c-40e2-81b0-625f59cf6395\",\n  \"name\": \"671489be-ab4c-40e2-81b0-625f59cf6395\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:22:24.890658519Z\",\n  \"endTime\": \"2026-03-30T06:22:55.769894295Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0011f0ed0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/be562cea-8623-4678-adaa-0057ff53bbd9",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "8b2df379-7ba3-4320-8e92-21a566e40b6c",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: BC420E598D454882AFE3F4518AA17A00 Ref B: CHI261140512052 Ref C: 2026-03-30T06:22:55Z",
                        ],
                        "Content-Length": ["503"],
                        "X-Ms-Request-Id": [
                            "fef2e839-309a-4189-8dfd-f18f57fe8ccf",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Rema...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

06:09:05 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapError | 0xc00130dda0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-zwkrrf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a",
      "name": "f32c7b79-2d68-46c9-a087-244c9417fc9a",
      "status": "Failed",
      "startTime": "2026-03-30T06:26:33.891991698Z",
      "endTime": "2026-03-30T06:27:03.911904895Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-zwkrrf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a\",\n  \"name\": \"f32c7b79-2d68-46c9-a087-244c9417fc9a\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:26:33.891991698Z\",\n  \"endTime\": \"2026-03-30T06:27:03.911904895Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00130dd80>{
            msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-zwkrrf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a\",\n  \"name\": \"f32c7b79-2d68-46c9-a087-244c9417fc9a\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:26:33.891991698Z\",\n  \"endTime\": \"2026-03-30T06:27:03.911904895Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000d77380>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Content-Type-Options": ["nosniff"],
                        "Date": [
                            "Mon, 30 Mar 2026 06:27:04 GMT",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/8159515f-155b-4e28-bb52-27a342aff475",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T062704Z:0a3e91fb-bed7-4520-9179-de2ded4f9ad5",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "cb801ab9-8e3a-4550-9150-d3ab4ebedcb1",
                        ],
                        "Strict-Transport-Security": [
   ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapError | 0xc00130dda0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-zwkrrf" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a",
      "name": "f32c7b79-2d68-46c9-a087-244c9417fc9a",
      "status": "Failed",
      "startTime": "2026-03-30T06:26:33.891991698Z",
      "endTime": "2026-03-30T06:27:03.911904895Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-zwkrrf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a\",\n  \"name\": \"f32c7b79-2d68-46c9-a087-244c9417fc9a\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:26:33.891991698Z\",\n  \"endTime\": \"2026-03-30T06:27:03.911904895Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00130dd80>{
            msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-zwkrrf\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/f32c7b79-2d68-46c9-a087-244c9417fc9a\",\n  \"name\": \"f32c7b79-2d68-46c9-a087-244c9417fc9a\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:26:33.891991698Z\",\n  \"endTime\": \"2026-03-30T06:27:03.911904895Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000d77380>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Content-Type-Options": ["nosniff"],
                        "Date": [
                            "Mon, 30 Mar 2026 06:27:04 GMT",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/8159515f-155b-4e28-bb52-27a342aff475",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T062704Z:0a3e91fb-bed7-4520-9179-de2ded4f9ad5",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "cb801ab9-8e3a-4550-9150-d3ab4ebedcb1",
                        ],
                        "Strict-Transport-Security": [
   ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Interrupted by User

06:09:05 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:35]: Interrupted by User

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

06:09:05 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapError | 0xc000c0e020>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-q6f5dn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a",
      "name": "ddad76d1-5b78-4fcc-8952-2fc6b191098a",
      "status": "Failed",
      "startTime": "2026-03-30T06:24:34.053136626Z",
      "endTime": "2026-03-30T06:25:20.835964185Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-q6f5dn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a\",\n  \"name\": \"ddad76d1-5b78-4fcc-8952-2fc6b191098a\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:24:34.053136626Z\",\n  \"endTime\": \"2026-03-30T06:25:20.835964185Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000c0e000>{
            msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-q6f5dn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a\",\n  \"name\": \"ddad76d1-5b78-4fcc-8952-2fc6b191098a\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:24:34.053136626Z\",\n  \"endTime\": \"2026-03-30T06:25:20.835964185Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0011a6f90>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "25de98a2-737b-4e00-a318-ad55c3f7864f",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/476c285c-9787-44a3-96f6-2cde38d0c586",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "Expires": ["-1"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T062525Z:b4a6cd2c-a7ff-...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapError | 0xc000c0e020>: 
    failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-q6f5dn" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a",
      "name": "ddad76d1-5b78-4fcc-8952-2fc6b191098a",
      "status": "Failed",
      "startTime": "2026-03-30T06:24:34.053136626Z",
      "endTime": "2026-03-30T06:25:20.835964185Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster hcp-cluster-np-128: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-q6f5dn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a\",\n  \"name\": \"ddad76d1-5b78-4fcc-8952-2fc6b191098a\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:24:34.053136626Z\",\n  \"endTime\": \"2026-03-30T06:25:20.835964185Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000c0e000>{
            msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-q6f5dn\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ddad76d1-5b78-4fcc-8952-2fc6b191098a\",\n  \"name\": \"ddad76d1-5b78-4fcc-8952-2fc6b191098a\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:24:34.053136626Z\",\n  \"endTime\": \"2026-03-30T06:25:20.835964185Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0011a6f90>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "25de98a2-737b-4e00-a318-ad55c3f7864f",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/476c285c-9787-44a3-96f6-2cde38d0c586",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "Expires": ["-1"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T062525Z:b4a6cd2c-a7ff-...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

06:09:05 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:100]: Unexpected error:
    <*fmt.wrapError | 0xc000c03700>: 
    failed to create HCP cluster cluster-zstream-4-20-8kr6g4: failed waiting for cluster="cluster-zstream-4-20-8kr6g4" in resourcegroup="rg-zstream-upgrade-4-20-m22d4f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569",
      "name": "6855e468-1276-4eca-9790-907f9d976569",
      "status": "Failed",
      "startTime": "2026-03-30T06:25:33.698883153Z",
      "endTime": "2026-03-30T06:26:01.513394715Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-20-8kr6g4: failed waiting for cluster=\"cluster-zstream-4-20-8kr6g4\" in resourcegroup=\"rg-zstream-upgrade-4-20-m22d4f\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569\",\n  \"name\": \"6855e468-1276-4eca-9790-907f9d976569\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:25:33.698883153Z\",\n  \"endTime\": \"2026-03-30T06:26:01.513394715Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000c036e0>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-20-8kr6g4\" in resourcegroup=\"rg-zstream-upgrade-4-20-m22d4f\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569\",\n  \"name\": \"6855e468-1276-4eca-9790-907f9d976569\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:25:33.698883153Z\",\n  \"endTime\": \"2026-03-30T06:26:01.513394715Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0013690e0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Correlation-Request-Id": [
                            "11f4df43-a99c-4d0b-85cc-765a6c02b47f",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T062604Z:11f4df43-a99c-4d0b-85cc-765a6c02b47f",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: D55D7FFF469D4C5DA1B7711B27E2352B Ref B: CHI261140509025 Ref C: 2026-03-30T06:26:04Z",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Ms-Request-Id": [
                            "4544b9fd-9af5-4ef6-ad11-8a7cc26e6366",
                        ],
                        "X-Envoy-Upstream-Service-Time": ["11"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "Content-Le...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:100]: Unexpected error:
    <*fmt.wrapError | 0xc000c03700>: 
    failed to create HCP cluster cluster-zstream-4-20-8kr6g4: failed waiting for cluster="cluster-zstream-4-20-8kr6g4" in resourcegroup="rg-zstream-upgrade-4-20-m22d4f" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569",
      "name": "6855e468-1276-4eca-9790-907f9d976569",
      "status": "Failed",
      "startTime": "2026-03-30T06:25:33.698883153Z",
      "endTime": "2026-03-30T06:26:01.513394715Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-20-8kr6g4: failed waiting for cluster=\"cluster-zstream-4-20-8kr6g4\" in resourcegroup=\"rg-zstream-upgrade-4-20-m22d4f\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569\",\n  \"name\": \"6855e468-1276-4eca-9790-907f9d976569\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:25:33.698883153Z\",\n  \"endTime\": \"2026-03-30T06:26:01.513394715Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000c036e0>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-20-8kr6g4\" in resourcegroup=\"rg-zstream-upgrade-4-20-m22d4f\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6855e468-1276-4eca-9790-907f9d976569\",\n  \"name\": \"6855e468-1276-4eca-9790-907f9d976569\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:25:33.698883153Z\",\n  \"endTime\": \"2026-03-30T06:26:01.513394715Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0013690e0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Correlation-Request-Id": [
                            "11f4df43-a99c-4d0b-85cc-765a6c02b47f",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T062604Z:11f4df43-a99c-4d0b-85cc-765a6c02b47f",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: D55D7FFF469D4C5DA1B7711B27E2352B Ref B: CHI261140509025 Ref C: 2026-03-30T06:26:04Z",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Ms-Request-Id": [
                            "4544b9fd-9af5-4ef6-ad11-8a7cc26e6366",
                        ],
                        "X-Envoy-Upstream-Service-Time": ["11"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "Content-Le...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

failed creating deployment "test-vm" in resourcegroup="<resource-group>": PUT <url>

06:09:05 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:114]: Timed out after 1800.001s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc0012c8180>: 
    failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-s5czh5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-s5czh5/providers/Microsoft.Resources/deployments/test-vm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidTemplateDeployment
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidTemplateDeployment",
        "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.",
        "details": [
          {
            "code": "ResourceCountExceedsLimitDueToTemplate",
            "message": "Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress",
            "details": []
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed creating deployment \"test-vm\" in resourcegroup=\"e2e-cidr-connectivity-s5czh5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-s5czh5/providers/Microsoft.Resources/deployments/test-vm\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplateDeployment\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidTemplateDeployment\",\n    \"message\": \"The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.\",\n    \"details\": [\n      {\n        \"code\": \"ResourceCountExceedsLimitDueToTemplate\",\n        \"message\": \"Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress\",\n        \"details\": []\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000f127e0>{
            ErrorCode: "InvalidTemplateDeployment",
            StatusCode: 400,
            RawResponse: {
                Status: "400 Bad Request",
                StatusCode: 400,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["198"],
                    "X-Ms-Failure-Cause": ["gateway"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2998"],
                    "X-Ms-Request-Id": [
                        "cf5d91a3-a913-4d5b-9164-bccf81eb8eee",
                    ],
                    "X-Ms-Correlation-Request-Id": [
                        "cf5d91a3-a913-4d5b-9164-bccf81eb8eee",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "EASTUS2EUAP:20260330T065415Z:cf5d91a3-a913-4d5b-9164-bccf81eb8eee",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "Date": [
                        "Mon, 30 Mar 2026 06:54:14 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Pragma": ["no-cache"],
                    "Expires": ["-1"],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Msedge-Ref": [
                        "Ref A: FED97ACCAC0E49C1AB5718A871C140C7 Ref B: CHI261140514023 Ref C: 2026-03-30T06:54:14Z",
                    ],
                    "Content-Length": ["655"],
                    "Content-Type": [
                        "application/json; charset=utf-8",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0012c8040>{
                    s: "{\"error\":{\"code\":\"InvalidTemplateDeployment\",\"message\":\"The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.\",\"details\":[{\"code\":\"ResourceCountExceedsLimitDueToTemplate\",\"message\":\"Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress\",\"detai...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:114]: Timed out after 1800.001s.
Expected success, but got an error:
    <*fmt.wrapError | 0xc0012c8180>: 
    failed creating deployment "test-vm" in resourcegroup="e2e-cidr-connectivity-s5czh5": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-s5czh5/providers/Microsoft.Resources/deployments/test-vm
    --------------------------------------------------------------------------------
    RESPONSE 400: 400 Bad Request
    ERROR CODE: InvalidTemplateDeployment
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "InvalidTemplateDeployment",
        "message": "The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.",
        "details": [
          {
            "code": "ResourceCountExceedsLimitDueToTemplate",
            "message": "Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress",
            "details": []
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed creating deployment \"test-vm\" in resourcegroup=\"e2e-cidr-connectivity-s5czh5\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/e2e-cidr-connectivity-s5czh5/providers/Microsoft.Resources/deployments/test-vm\n--------------------------------------------------------------------------------\nRESPONSE 400: 400 Bad Request\nERROR CODE: InvalidTemplateDeployment\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"InvalidTemplateDeployment\",\n    \"message\": \"The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.\",\n    \"details\": [\n      {\n        \"code\": \"ResourceCountExceedsLimitDueToTemplate\",\n        \"message\": \"Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress\",\n        \"details\": []\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*exported.ResponseError | 0xc000f127e0>{
            ErrorCode: "InvalidTemplateDeployment",
            StatusCode: 400,
            RawResponse: {
                Status: "400 Bad Request",
                StatusCode: 400,
                Proto: "HTTP/2.0",
                ProtoMajor: 2,
                ProtoMinor: 0,
                Header: {
                    "X-Ms-Ratelimit-Remaining-Subscription-Writes": ["198"],
                    "X-Ms-Failure-Cause": ["gateway"],
                    "X-Ms-Ratelimit-Remaining-Subscription-Global-Writes": ["2998"],
                    "X-Ms-Request-Id": [
                        "cf5d91a3-a913-4d5b-9164-bccf81eb8eee",
                    ],
                    "X-Ms-Correlation-Request-Id": [
                        "cf5d91a3-a913-4d5b-9164-bccf81eb8eee",
                    ],
                    "X-Ms-Routing-Request-Id": [
                        "EASTUS2EUAP:20260330T065415Z:cf5d91a3-a913-4d5b-9164-bccf81eb8eee",
                    ],
                    "X-Content-Type-Options": ["nosniff"],
                    "Date": [
                        "Mon, 30 Mar 2026 06:54:14 GMT",
                    ],
                    "Cache-Control": ["no-cache"],
                    "Pragma": ["no-cache"],
                    "Expires": ["-1"],
                    "Strict-Transport-Security": [
                        "max-age=31536000; includeSubDomains",
                    ],
                    "X-Msedge-Ref": [
                        "Ref A: FED97ACCAC0E49C1AB5718A871C140C7 Ref B: CHI261140514023 Ref C: 2026-03-30T06:54:14Z",
                    ],
                    "Content-Length": ["655"],
                    "Content-Type": [
                        "application/json; charset=utf-8",
                    ],
                    "X-Cache": ["CONFIG_NOCACHE"],
                },
                Body: <*exported.nopClosingBytesReader | 0xc0012c8040>{
                    s: "{\"error\":{\"code\":\"InvalidTemplateDeployment\",\"message\":\"The template deployment 'test-vm' is not valid according to the validation procedure. The tracking id is 'cf5d91a3-a913-4d5b-9164-bccf81eb8eee'. See inner errors for details.\",\"details\":[{\"code\":\"ResourceCountExceedsLimitDueToTemplate\",\"message\":\"Subscription XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX has a quota of 20 for resources of type PublicIpAddress with sku SkuNotSpecified. Subscription currently has 20 resources and the template contains 1 new resources of the this type which exceeds the quota. Please contact support to increase the quota for resource type PublicIpAddress\",\"detai...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

06:09:05 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:100]: Unexpected error:
    <*fmt.wrapError | 0xc00068d580>: 
    failed to create HCP cluster cluster-zstream-4-21-wggw2c: failed waiting for cluster="cluster-zstream-4-21-wggw2c" in resourcegroup="rg-zstream-upgrade-4-21-6hp9ww" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029",
      "name": "d05a5529-ae05-46fe-83bf-87a75636e029",
      "status": "Failed",
      "startTime": "2026-03-30T06:26:33.107623699Z",
      "endTime": "2026-03-30T06:27:03.896795282Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-21-wggw2c: failed waiting for cluster=\"cluster-zstream-4-21-wggw2c\" in resourcegroup=\"rg-zstream-upgrade-4-21-6hp9ww\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029\",\n  \"name\": \"d05a5529-ae05-46fe-83bf-87a75636e029\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:26:33.107623699Z\",\n  \"endTime\": \"2026-03-30T06:27:03.896795282Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00068d560>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-21-wggw2c\" in resourcegroup=\"rg-zstream-upgrade-4-21-6hp9ww\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029\",\n  \"name\": \"d05a5529-ae05-46fe-83bf-87a75636e029\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:26:33.107623699Z\",\n  \"endTime\": \"2026-03-30T06:27:03.896795282Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0012c8bd0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "4e624f15-373d-4b5d-9aa2-5728bfa533ba",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/ddf1cb1e-19cd-44b1-a6fc-7ee87af76e16",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"],
                        "X-Ms-Correlation-Request-Id": [
                            "9f65d160-79d1-46b3-8b9c-23b37e9e9527",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: E48A2BDBE5144E9E9D51043FBB540576 Ref B: CHI261140513040 Ref C: 2026-03-30T06:27:04Z",
                        ],
      ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:100]: Unexpected error:
    <*fmt.wrapError | 0xc00068d580>: 
    failed to create HCP cluster cluster-zstream-4-21-wggw2c: failed waiting for cluster="cluster-zstream-4-21-wggw2c" in resourcegroup="rg-zstream-upgrade-4-21-6hp9ww" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029",
      "name": "d05a5529-ae05-46fe-83bf-87a75636e029",
      "status": "Failed",
      "startTime": "2026-03-30T06:26:33.107623699Z",
      "endTime": "2026-03-30T06:27:03.896795282Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-21-wggw2c: failed waiting for cluster=\"cluster-zstream-4-21-wggw2c\" in resourcegroup=\"rg-zstream-upgrade-4-21-6hp9ww\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029\",\n  \"name\": \"d05a5529-ae05-46fe-83bf-87a75636e029\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:26:33.107623699Z\",\n  \"endTime\": \"2026-03-30T06:27:03.896795282Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00068d560>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-21-wggw2c\" in resourcegroup=\"rg-zstream-upgrade-4-21-6hp9ww\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d05a5529-ae05-46fe-83bf-87a75636e029\",\n  \"name\": \"d05a5529-ae05-46fe-83bf-87a75636e029\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:26:33.107623699Z\",\n  \"endTime\": \"2026-03-30T06:27:03.896795282Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0012c8bd0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "4e624f15-373d-4b5d-9aa2-5728bfa533ba",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/ddf1cb1e-19cd-44b1-a6fc-7ee87af76e16",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"],
                        "X-Ms-Correlation-Request-Id": [
                            "9f65d160-79d1-46b3-8b9c-23b37e9e9527",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: E48A2BDBE5144E9E9D51043FBB540576 Ref B: CHI261140513040 Ref C: 2026-03-30T06:27:04Z",
                        ],
      ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

06:09:05 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapError | 0xc0011eebe0>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-4rvwkj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91",
      "name": "db0f712c-aa43-42f2-ba74-2b516fee6d91",
      "status": "Failed",
      "startTime": "2026-03-30T06:20:12.516389794Z",
      "endTime": "2026-03-30T06:20:48.848822062Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4rvwkj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91\",\n  \"name\": \"db0f712c-aa43-42f2-ba74-2b516fee6d91\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:20:12.516389794Z\",\n  \"endTime\": \"2026-03-30T06:20:48.848822062Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0011eebc0>{
            msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4rvwkj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91\",\n  \"name\": \"db0f712c-aa43-42f2-ba74-2b516fee6d91\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:20:12.516389794Z\",\n  \"endTime\": \"2026-03-30T06:20:48.848822062Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0015993e0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "783c170b-8f66-4831-be6b-bc51973e70b7",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/d9f54903-b7b1-4721-9c43-2cc2e56ffb60",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"],
                        "Pragma": ["no-cache"],
                        "Content-Length": ["503"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"],
                        "Cache-Control": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Correlation-Request-Id": [
                            "44d4e8c9-3387-4e0a-b086-ecf8fbc2f225",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T062053Z:44d4e8c9-3387-4e0a-b086-ecf8fbc2f225",
                        ],
 ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapError | 0xc0011eebe0>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-4rvwkj" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91",
      "name": "db0f712c-aa43-42f2-ba74-2b516fee6d91",
      "status": "Failed",
      "startTime": "2026-03-30T06:20:12.516389794Z",
      "endTime": "2026-03-30T06:20:48.848822062Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4rvwkj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91\",\n  \"name\": \"db0f712c-aa43-42f2-ba74-2b516fee6d91\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:20:12.516389794Z\",\n  \"endTime\": \"2026-03-30T06:20:48.848822062Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0011eebc0>{
            msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-4rvwkj\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/db0f712c-aa43-42f2-ba74-2b516fee6d91\",\n  \"name\": \"db0f712c-aa43-42f2-ba74-2b516fee6d91\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:20:12.516389794Z\",\n  \"endTime\": \"2026-03-30T06:20:48.848822062Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0015993e0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "783c170b-8f66-4831-be6b-bc51973e70b7",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/d9f54903-b7b1-4721-9c43-2cc2e56ffb60",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"],
                        "Pragma": ["no-cache"],
                        "Content-Length": ["503"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"],
                        "Cache-Control": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Correlation-Request-Id": [
                            "44d4e8c9-3387-4e0a-b086-ecf8fbc2f225",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T062053Z:44d4e8c9-3387-4e0a-b086-ecf8fbc2f225",
                        ],
 ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

06:09:05 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
    <*fmt.wrapError | 0xc000c0fd00>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-lsx2ks" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279",
      "name": "0f618886-6ee0-48de-9547-a72389e82279",
      "status": "Failed",
      "startTime": "2026-03-30T06:18:28.258152765Z",
      "endTime": "2026-03-30T06:18:44.003768858Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lsx2ks\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279\",\n  \"name\": \"0f618886-6ee0-48de-9547-a72389e82279\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:18:28.258152765Z\",\n  \"endTime\": \"2026-03-30T06:18:44.003768858Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000c0fc60>{
            msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lsx2ks\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279\",\n  \"name\": \"0f618886-6ee0-48de-9547-a72389e82279\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:18:28.258152765Z\",\n  \"endTime\": \"2026-03-30T06:18:44.003768858Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00106a720>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Content-Type-Options": ["nosniff"],
                        "Date": [
                            "Mon, 30 Mar 2026 06:18:48 GMT",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Correlation-Request-Id": [
                            "022cb22e-6859-4c7f-a168-ebde742732c9",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["503"],
                        "Expires": ["-1"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T061849Z:022cb22e-6859-4c7f-a168-ebde742732c9",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
    <*fmt.wrapError | 0xc000c0fd00>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-lsx2ks" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279",
      "name": "0f618886-6ee0-48de-9547-a72389e82279",
      "status": "Failed",
      "startTime": "2026-03-30T06:18:28.258152765Z",
      "endTime": "2026-03-30T06:18:44.003768858Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lsx2ks\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279\",\n  \"name\": \"0f618886-6ee0-48de-9547-a72389e82279\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:18:28.258152765Z\",\n  \"endTime\": \"2026-03-30T06:18:44.003768858Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000c0fc60>{
            msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-lsx2ks\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f618886-6ee0-48de-9547-a72389e82279\",\n  \"name\": \"0f618886-6ee0-48de-9547-a72389e82279\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:18:28.258152765Z\",\n  \"endTime\": \"2026-03-30T06:18:44.003768858Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00106a720>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Content-Type-Options": ["nosniff"],
                        "Date": [
                            "Mon, 30 Mar 2026 06:18:48 GMT",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Correlation-Request-Id": [
                            "022cb22e-6859-4c7f-a168-ebde742732c9",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["503"],
                        "Expires": ["-1"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260330T061849Z:022cb22e-6859-4c7f-a168-ebde742732c9",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

06:09:05 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc00160a6f0>: 
    failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-zd6bdf" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-zd6bdf\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0015092b0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf",
            },
            <*fmt.wrapErrors | 0xc00160a6c0>{
                msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-zd6bdf\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0015092b0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc00160a6f0>: 
    failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-zd6bdf" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-zd6bdf\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0015092b0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf",
            },
            <*fmt.wrapErrors | 0xc00160a6c0>{
                msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-zd6bdf\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0015092b0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-zd6bdf",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

06:09:05 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: Unexpected error:
    <*fmt.wrapError | 0xc0016131a0>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-8qw9d2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e",
      "name": "e8c3c061-19f1-4d72-b474-3fbe98891e9e",
      "status": "Failed",
      "startTime": "2026-03-30T06:18:33.483509413Z",
      "endTime": "2026-03-30T06:18:44.018298845Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8qw9d2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n  \"name\": \"e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:18:33.483509413Z\",\n  \"endTime\": \"2026-03-30T06:18:44.018298845Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc001613180>{
            msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8qw9d2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n  \"name\": \"e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:18:33.483509413Z\",\n  \"endTime\": \"2026-03-30T06:18:44.018298845Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00157d260>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Routing-Request-Id": [
                            "CANADAEAST:20260330T061854Z:72e130e2-01fe-4e82-86ac-3810c7b0b18a",
                        ],
                        "Date": [
                            "Mon, 30 Mar 2026 06:18:53 GMT",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "72e130e2-01fe-4e82-86ac-3810c7b0b18a",
                        ],
                        "Pragma": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: 05A3F815056F4198B1B5D4420C306AED Ref B: CHI261140504023 Ref C: 2026-03-30T06:18:53Z",
                        ],
                        "Cache-Control": ["no-cache"],
                        "X-Ms-Request-Id": [
                            "111c62...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: Unexpected error:
    <*fmt.wrapError | 0xc0016131a0>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-8qw9d2" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e",
      "name": "e8c3c061-19f1-4d72-b474-3fbe98891e9e",
      "status": "Failed",
      "startTime": "2026-03-30T06:18:33.483509413Z",
      "endTime": "2026-03-30T06:18:44.018298845Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8qw9d2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n  \"name\": \"e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:18:33.483509413Z\",\n  \"endTime\": \"2026-03-30T06:18:44.018298845Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc001613180>{
            msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-8qw9d2\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n  \"name\": \"e8c3c061-19f1-4d72-b474-3fbe98891e9e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-30T06:18:33.483509413Z\",\n  \"endTime\": \"2026-03-30T06:18:44.018298845Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00157d260>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Routing-Request-Id": [
                            "CANADAEAST:20260330T061854Z:72e130e2-01fe-4e82-86ac-3810c7b0b18a",
                        ],
                        "Date": [
                            "Mon, 30 Mar 2026 06:18:53 GMT",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "72e130e2-01fe-4e82-86ac-3810c7b0b18a",
                        ],
                        "Pragma": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: 05A3F815056F4198B1B5D4420C306AED Ref B: CHI261140504023 Ref C: 2026-03-30T06:18:53Z",
                        ],
                        "Cache-Control": ["no-cache"],
                        "X-Ms-Request-Id": [
                            "111c62...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Environment: PROD