CIHealth Run Log

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
23:56:06	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	unknown	failed	n/a	1	non-artifact-backed failure (no junit artifacts)
21:39:24	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	22	e2e (22) ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 21:39:24 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error: <fmt.wrapError \| 0xc0002e4c80>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-bnf7qs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c", "name": "a7266eb7-02fe-4e25-8a6b-80bf2b07af6c", "status": "Failed", "startTime": "2026-03-18T22:04:06.780626426Z", "endTime": "2026-03-18T22:04:50.639636342Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-bnf7qs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\",\n \"name\": \"a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:04:06.780626426Z\",\n \"endTime\": \"2026-03-18T22:04:50.639636342Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0002e4c60>{ msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-bnf7qs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\",\n \"name\": \"a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:04:06.780626426Z\",\n \"endTime\": \"2026-03-18T22:04:50.639636342Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001556900>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "68337901-b202-4387-8b62-a5d5fa9a8a87", ], "Pragma": ["no-cache"], "Expires": ["-1"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/86f586e0-8021-49ae-8dab-791e7e4ddc69", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Correlation-Request-Id": [ "b2f55a6e-086e-4aea-93f6-026a3332b1ba", ], "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: 340E61C37B254A58B9B02493846AE154 Ref B: CHI261140503054 Ref C: 2026-03-18T22:04:57Z", ], "Content-Type": ["application/json"], "X-Ms-Ratelimit-Rem... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error: <fmt.wrapError \| 0xc0002e4c80>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-bnf7qs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c", "name": "a7266eb7-02fe-4e25-8a6b-80bf2b07af6c", "status": "Failed", "startTime": "2026-03-18T22:04:06.780626426Z", "endTime": "2026-03-18T22:04:50.639636342Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-bnf7qs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\",\n \"name\": \"a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:04:06.780626426Z\",\n \"endTime\": \"2026-03-18T22:04:50.639636342Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0002e4c60>{ msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-bnf7qs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\",\n \"name\": \"a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:04:06.780626426Z\",\n \"endTime\": \"2026-03-18T22:04:50.639636342Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001556900>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "68337901-b202-4387-8b62-a5d5fa9a8a87", ], "Pragma": ["no-cache"], "Expires": ["-1"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/86f586e0-8021-49ae-8dab-791e7e4ddc69", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Correlation-Request-Id": [ "b2f55a6e-086e-4aea-93f6-026a3332b1ba", ], "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: 340E61C37B254A58B9B02493846AE154 Ref B: CHI261140503054 Ref C: 2026-03-18T22:04:57Z", ], "Content-Type": ["application/json"], "X-Ms-Ratelimit-Rem... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 21:39:24 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error: <fmt.wrapErrors \| 0xc00101e2d0>: failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nxd528" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded { msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nxd528\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001374460>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528", }, <fmt.wrapErrors \| 0xc00101e2a0>{ msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nxd528\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001374460>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error: <fmt.wrapErrors \| 0xc00101e2d0>: failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nxd528" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded { msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nxd528\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001374460>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528", }, <fmt.wrapErrors \| 0xc00101e2a0>{ msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nxd528\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001374460>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528", }, <context.deadlineExceededError>{}, ], }, ], } occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 21:39:24 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error: <fmt.wrapErrors \| 0xc000e6cb10>: failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-vm79bx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded { msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vm79bx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0011648c0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx", }, <fmt.wrapErrors \| 0xc000e6cae0>{ msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vm79bx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0011648c0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error: <fmt.wrapErrors \| 0xc000e6cb10>: failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-vm79bx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded { msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vm79bx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0011648c0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx", }, <fmt.wrapErrors \| 0xc000e6cae0>{ msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vm79bx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0011648c0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx", }, <context.deadlineExceededError>{}, ], }, ], } occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 21:39:24 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error: <fmt.wrapErrors \| 0xc000fc8e70>: failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-zdx4fc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: context deadline exceeded { msg: "failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-zdx4fc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0011f5360>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc", }, <fmt.wrapErrors \| 0xc000fc8e40>{ msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-zdx4fc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0011f5360>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error: <fmt.wrapErrors \| 0xc000fc8e70>: failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-zdx4fc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: context deadline exceeded { msg: "failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-zdx4fc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0011f5360>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc", }, <fmt.wrapErrors \| 0xc000fc8e40>{ msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-zdx4fc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0011f5360>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc", }, <context.deadlineExceededError>{}, ], }, ], } occurred Deserializaion Error: no output from command 21:39:24 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel Show raw failure > github.com/Azure/ARO-HCP/test/e2e.init.func15.2({0x7f54ec580e60, 0xc0009fb7d0}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126 \| \| By("waiting for first cluster to complete creation") > _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{ \| Frequency: framework.StandardPollInterval, \| }) github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x827b940?, 0xc0009fb7d0?}) /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 github.com/onsi/ginkgo/v2/internal.(Suite).runNode.func3() /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 github.com/onsi/ginkgo/v2/internal.(Suite).runNode in goroutine 84 /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 ------------------------------ [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 03/18/26 23:20:55.083 "ts"="2026-03-18 23:20:55.083532" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-18 23:20:55.083598" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-kjjpjb" "resourceGroup"="customer-rg-2hmq5s" "ts"="2026-03-18 23:20:55.333961" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-kjjpjb" "resourceGroup"="" "ts"="2026-03-18 23:20:56.369508" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-wlxsb8" "resourceGroup"="customer-rg-2hmq5s" "ts"="2026-03-18 23:20:56.519635" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-wlxsb8" "resourceGroup"="" "ts"="2026-03-18 23:20:57.115820" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz" "ts"="2026-03-18 23:20:57.115955" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz" "ts"="2026-03-18 23:20:57.116022" "level"=0 "msg"="collecting debug info" "ts"="2026-03-18 23:20:57.117126" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-26" "ts"="2026-03-18 23:20:57.117146" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-110" "ts"="2026-03-18 23:20:57.117168" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-2hmq5s" "ts"="2026-03-18 23:20:57.117208" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-36" "ts"="2026-03-18 23:20:57.264846" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-110" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-18 23:20:57.327570" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-36" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-18 23:20:57.396893" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-110" "deployment"="msi-deployment-110" "ts"="2026-03-18 23:20:57.432057" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32" "ts"="2026-03-18 23:20:57.548409" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-36" "deployment"="msi-deployment-36" "ts"="2026-03-18 23:20:57.629848" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="customer-infra-basic-hcp-cluster2-wlxsb8" "ts"="2026-03-18 23:20:57.773291" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-26" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-18 23:20:57.819000" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4" "ts"="2026-03-18 23:20:57.904929" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="customer-infra-basic-hcp-cluster-kjjpjb" "ts"="2026-03-18 23:20:58.268158" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-26" "deployment"="msi-deployment-26" "ts"="2026-03-18 23:20:58.435912" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-18 23:20:58.436063" "level"=0 "msg"="deleting created resources" "ts"="2026-03-18 23:20:58.436192" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-2hmq5s" Command Error: signal: killed Deserializaion Error: no output from command > github.com/Azure/ARO-HCP/test/e2e.init.func15.2({0x7f54ec580e60, 0xc0009fb7d0}) /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126 \| \| By("waiting for first cluster to complete creation") > _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{ \| Frequency: framework.StandardPollInterval, \| }) github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x827b940?, 0xc0009fb7d0?}) /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465 github.com/onsi/ginkgo/v2/internal.(Suite).runNode.func3() /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901 github.com/onsi/ginkgo/v2/internal.(Suite).runNode in goroutine 84 /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888 ------------------------------ [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 03/18/26 23:20:55.083 "ts"="2026-03-18 23:20:55.083532" "level"=0 "msg"="Commiting timing metadata." "ts"="2026-03-18 23:20:55.083598" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-kjjpjb" "resourceGroup"="customer-rg-2hmq5s" "ts"="2026-03-18 23:20:55.333961" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-kjjpjb" "resourceGroup"="" "ts"="2026-03-18 23:20:56.369508" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-wlxsb8" "resourceGroup"="customer-rg-2hmq5s" "ts"="2026-03-18 23:20:56.519635" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-wlxsb8" "resourceGroup"="" "ts"="2026-03-18 23:20:57.115820" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz" "ts"="2026-03-18 23:20:57.115955" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz" "ts"="2026-03-18 23:20:57.116022" "level"=0 "msg"="collecting debug info" "ts"="2026-03-18 23:20:57.117126" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-26" "ts"="2026-03-18 23:20:57.117146" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-110" "ts"="2026-03-18 23:20:57.117168" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-2hmq5s" "ts"="2026-03-18 23:20:57.117208" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-36" "ts"="2026-03-18 23:20:57.264846" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-110" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-18 23:20:57.327570" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-36" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-18 23:20:57.396893" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-110" "deployment"="msi-deployment-110" "ts"="2026-03-18 23:20:57.432057" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32" "ts"="2026-03-18 23:20:57.548409" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-36" "deployment"="msi-deployment-36" "ts"="2026-03-18 23:20:57.629848" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="customer-infra-basic-hcp-cluster2-wlxsb8" "ts"="2026-03-18 23:20:57.773291" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-26" "deployment"="pooledMsiScopedAssignments" "ts"="2026-03-18 23:20:57.819000" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4" "ts"="2026-03-18 23:20:57.904929" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="customer-infra-basic-hcp-cluster-kjjpjb" "ts"="2026-03-18 23:20:58.268158" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-26" "deployment"="msi-deployment-26" "ts"="2026-03-18 23:20:58.435912" "level"=0 "msg"="finished collecting debug info" "ts"="2026-03-18 23:20:58.436063" "level"=0 "msg"="deleting created resources" "ts"="2026-03-18 23:20:58.436192" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-2hmq5s" Command Error: signal: killed Deserializaion Error: no output from command timeout during CreateHCPClusterAndWait; context deadline exceeded 21:39:24 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error: <fmt.wrapErrors \| 0xc0014de840>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-txz9bx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-txz9bx, error: context deadline exceeded { msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-txz9bx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-txz9bx, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000c651e0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-txz9bx", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error: <fmt.wrapErrors \| 0xc0014de840>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-txz9bx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-txz9bx, error: context deadline exceeded { msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-txz9bx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-txz9bx, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000c651e0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-txz9bx", }, <context.deadlineExceededError>{}, ], } occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 21:39:24 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc0009001a0>: failed to create HCP cluster cluster-zstream-4-21-94f5n8: failed waiting for cluster="cluster-zstream-4-21-94f5n8" in resourcegroup="rg-zstream-upgrade-4-21-64j8h4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1", "name": "8cac1b2e-d936-402c-86be-44b7a37d4ae1", "status": "Failed", "startTime": "2026-03-18T22:09:58.306382688Z", "endTime": "2026-03-18T22:11:58.452014263Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-94f5n8: failed waiting for cluster=\"cluster-zstream-4-21-94f5n8\" in resourcegroup=\"rg-zstream-upgrade-4-21-64j8h4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1\",\n \"name\": \"8cac1b2e-d936-402c-86be-44b7a37d4ae1\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:09:58.306382688Z\",\n \"endTime\": \"2026-03-18T22:11:58.452014263Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000900180>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-94f5n8\" in resourcegroup=\"rg-zstream-upgrade-4-21-64j8h4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1\",\n \"name\": \"8cac1b2e-d936-402c-86be-44b7a37d4ae1\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:09:58.306382688Z\",\n \"endTime\": \"2026-03-18T22:11:58.452014263Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000575020>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260318T221200Z:cc127123-120b-4dba-b721-6d7a1221b757", ], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "88b27e98-71ad-423d-941a-3dfed915fe8e", ], "X-Ms-Correlation-Request-Id": [ "cc127123-120b-4dba-b721-6d7a1221b757", ], "Date": [ "Wed, 18 Mar 2026 22:12:00 GMT", ], "Content-Length": ["503"], "Expires": ["-1"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/f9182697-47ef-450e-985b-d1bab32508ff", ], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc0009001a0>: failed to create HCP cluster cluster-zstream-4-21-94f5n8: failed waiting for cluster="cluster-zstream-4-21-94f5n8" in resourcegroup="rg-zstream-upgrade-4-21-64j8h4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1", "name": "8cac1b2e-d936-402c-86be-44b7a37d4ae1", "status": "Failed", "startTime": "2026-03-18T22:09:58.306382688Z", "endTime": "2026-03-18T22:11:58.452014263Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-21-94f5n8: failed waiting for cluster=\"cluster-zstream-4-21-94f5n8\" in resourcegroup=\"rg-zstream-upgrade-4-21-64j8h4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1\",\n \"name\": \"8cac1b2e-d936-402c-86be-44b7a37d4ae1\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:09:58.306382688Z\",\n \"endTime\": \"2026-03-18T22:11:58.452014263Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000900180>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-94f5n8\" in resourcegroup=\"rg-zstream-upgrade-4-21-64j8h4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1\",\n \"name\": \"8cac1b2e-d936-402c-86be-44b7a37d4ae1\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:09:58.306382688Z\",\n \"endTime\": \"2026-03-18T22:11:58.452014263Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000575020>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260318T221200Z:cc127123-120b-4dba-b721-6d7a1221b757", ], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "88b27e98-71ad-423d-941a-3dfed915fe8e", ], "X-Ms-Correlation-Request-Id": [ "cc127123-120b-4dba-b721-6d7a1221b757", ], "Date": [ "Wed, 18 Mar 2026 22:12:00 GMT", ], "Content-Length": ["503"], "Expires": ["-1"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/f9182697-47ef-450e-985b-d1bab32508ff", ], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 21:39:24 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error: <fmt.wrapErrors \| 0xc000de6db0>: failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-fjpggg" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: context deadline exceeded { msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-fjpggg\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001575960>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg", }, <fmt.wrapErrors \| 0xc000de6d80>{ msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-fjpggg\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001575960>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error: <fmt.wrapErrors \| 0xc000de6db0>: failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-fjpggg" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: context deadline exceeded { msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-fjpggg\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001575960>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg", }, <fmt.wrapErrors \| 0xc000de6d80>{ msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-fjpggg\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001575960>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg", }, <context.deadlineExceededError>{}, ], }, ], } occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 21:39:24 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error: <fmt.wrapError \| 0xc000a02180>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-dngc6m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306", "name": "e959e2ef-e55d-46f1-8f37-18a399818306", "status": "Failed", "startTime": "2026-03-18T21:58:25.256129849Z", "endTime": "2026-03-18T21:59:35.785955056Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-dngc6m\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306\",\n \"name\": \"e959e2ef-e55d-46f1-8f37-18a399818306\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:25.256129849Z\",\n \"endTime\": \"2026-03-18T21:59:35.785955056Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a02160>{ msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-dngc6m\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306\",\n \"name\": \"e959e2ef-e55d-46f1-8f37-18a399818306\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:25.256129849Z\",\n \"endTime\": \"2026-03-18T21:59:35.785955056Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000978450>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Correlation-Request-Id": [ "6b4b037b-5eae-4b4a-89a0-993dc36429cf", ], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Wed, 18 Mar 2026 21:59:36 GMT", ], "Content-Type": ["application/json"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/7240e2d5-207e-4f6d-805f-097d383fb77f", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Routing-Request-Id": [ "CANADAEAST:20260318T215936Z:6b4b037b-5eae-4b4a-89a0-993dc36429cf", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Cache-Control": ["no-... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error: <fmt.wrapError \| 0xc000a02180>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-dngc6m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306", "name": "e959e2ef-e55d-46f1-8f37-18a399818306", "status": "Failed", "startTime": "2026-03-18T21:58:25.256129849Z", "endTime": "2026-03-18T21:59:35.785955056Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-dngc6m\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306\",\n \"name\": \"e959e2ef-e55d-46f1-8f37-18a399818306\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:25.256129849Z\",\n \"endTime\": \"2026-03-18T21:59:35.785955056Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a02160>{ msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-dngc6m\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306\",\n \"name\": \"e959e2ef-e55d-46f1-8f37-18a399818306\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:25.256129849Z\",\n \"endTime\": \"2026-03-18T21:59:35.785955056Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000978450>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Correlation-Request-Id": [ "6b4b037b-5eae-4b4a-89a0-993dc36429cf", ], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Wed, 18 Mar 2026 21:59:36 GMT", ], "Content-Type": ["application/json"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/7240e2d5-207e-4f6d-805f-097d383fb77f", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Routing-Request-Id": [ "CANADAEAST:20260318T215936Z:6b4b037b-5eae-4b4a-89a0-993dc36429cf", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Cache-Control": ["no-... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 21:39:24 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapError \| 0xc0009927a0>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-2x5tqd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c", "name": "15b94b31-6297-4e0a-818d-5ea9d03e264c", "status": "Failed", "startTime": "2026-03-18T21:58:09.542609231Z", "endTime": "2026-03-18T21:59:35.888371839Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-2x5tqd\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c\",\n \"name\": \"15b94b31-6297-4e0a-818d-5ea9d03e264c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:09.542609231Z\",\n \"endTime\": \"2026-03-18T21:59:35.888371839Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000992780>{ msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-2x5tqd\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c\",\n \"name\": \"15b94b31-6297-4e0a-818d-5ea9d03e264c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:09.542609231Z\",\n \"endTime\": \"2026-03-18T21:59:35.888371839Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000928b40>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Content-Length": ["503"], "X-Ms-Correlation-Request-Id": [ "85a78c11-e075-4480-8bc1-622a8ec78799", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: 5C0E582900C8470A8392AAB9D62C27B0 Ref B: CHI261140504040 Ref C: 2026-03-18T21:59:41Z", ], "Date": [ "Wed, 18 Mar 2026 21:59:40 GMT", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"], "Pragma": ["no-cache"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "ee2f9f9d-ea91-4be1-8440-27274248be3e", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error: <fmt.wrapError \| 0xc0009927a0>: failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-2x5tqd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c", "name": "15b94b31-6297-4e0a-818d-5ea9d03e264c", "status": "Failed", "startTime": "2026-03-18T21:58:09.542609231Z", "endTime": "2026-03-18T21:59:35.888371839Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-2x5tqd\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c\",\n \"name\": \"15b94b31-6297-4e0a-818d-5ea9d03e264c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:09.542609231Z\",\n \"endTime\": \"2026-03-18T21:59:35.888371839Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000992780>{ msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-2x5tqd\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c\",\n \"name\": \"15b94b31-6297-4e0a-818d-5ea9d03e264c\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:09.542609231Z\",\n \"endTime\": \"2026-03-18T21:59:35.888371839Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000928b40>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Content-Length": ["503"], "X-Ms-Correlation-Request-Id": [ "85a78c11-e075-4480-8bc1-622a8ec78799", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: 5C0E582900C8470A8392AAB9D62C27B0 Ref B: CHI261140504040 Ref C: 2026-03-18T21:59:41Z", ], "Date": [ "Wed, 18 Mar 2026 21:59:40 GMT", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"], "Pragma": ["no-cache"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "ee2f9f9d-ea91-4be1-8440-27274248be3e", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 21:39:24 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error: <fmt.wrapErrors \| 0xc00142ccf0>: failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-sjngpl" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded { msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-sjngpl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00158dc10>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl", }, <fmt.wrapErrors \| 0xc00142ccc0>{ msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-sjngpl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00158dc10>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error: <fmt.wrapErrors \| 0xc00142ccf0>: failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-sjngpl" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded { msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-sjngpl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00158dc10>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl", }, <fmt.wrapErrors \| 0xc00142ccc0>{ msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-sjngpl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc00158dc10>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl", }, <context.deadlineExceededError>{}, ], }, ], } occurred ERROR CODE: MultipleErrorsOccurred; detail code InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 21:39:24 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapError \| 0xc00133a3c0>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-sv2zcp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e", "name": "b06360bf-0a54-4e7d-b9f1-3e5b6626253e", "status": "Failed", "startTime": "2026-03-18T22:17:00.823479898Z", "endTime": "2026-03-18T22:18:09.152905722Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" }, { "code": "InternalServerError", "message": "Failed to get virtual network 'customer-vnet-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" }, { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-120/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" }, { "code": "InternalServerError", "message": "failed to get network security group 'customer-nsg-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" }, { "code": "InternalServerError", "message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-sv2zcp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n \"name\": \"b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:17:00.823479898Z\",\n \"endTime\": \"2026-03-18T22:18:09.152905722Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-120/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00133a3a0>{ msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-sv2zcp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n \"name\": \"b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:17:00.823479898Z\",\n \"endTime\": \"2026-03-18T22:18:09.152905722Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-name' details: Cl... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapError \| 0xc00133a3c0>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-sv2zcp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e", "name": "b06360bf-0a54-4e7d-b9f1-3e5b6626253e", "status": "Failed", "startTime": "2026-03-18T22:17:00.823479898Z", "endTime": "2026-03-18T22:18:09.152905722Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" }, { "code": "InternalServerError", "message": "Failed to get virtual network 'customer-vnet-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" }, { "code": "InternalServerError", "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-120/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" }, { "code": "InternalServerError", "message": "failed to get network security group 'customer-nsg-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" }, { "code": "InternalServerError", "message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-sv2zcp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n \"name\": \"b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:17:00.823479898Z\",\n \"endTime\": \"2026-03-18T22:18:09.152905722Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-120/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get network security group 'customer-nsg-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00133a3a0>{ msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-sv2zcp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n \"name\": \"b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:17:00.823479898Z\",\n \"endTime\": \"2026-03-18T22:18:09.152905722Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to get virtual network 'customer-vnet-name' details: Cl... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 21:39:24 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error: <fmt.wrapErrors \| 0xc0010cc570>: failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-jrpfvc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: context deadline exceeded { msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-jrpfvc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000fde070>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc", }, <fmt.wrapErrors \| 0xc0010cc540>{ msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-jrpfvc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000fde070>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error: <fmt.wrapErrors \| 0xc0010cc570>: failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-jrpfvc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: context deadline exceeded { msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-jrpfvc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000fde070>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc", }, <fmt.wrapErrors \| 0xc0010cc540>{ msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-jrpfvc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000fde070>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc", }, <context.deadlineExceededError>{}, ], }, ], } occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 21:39:24 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapError \| 0xc000a8e6a0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-fncbll" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b", "name": "1dcfdab9-27df-483f-a009-42303866394b", "status": "Failed", "startTime": "2026-03-18T22:10:00.698917174Z", "endTime": "2026-03-18T22:12:23.691383966Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-fncbll\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\",\n \"name\": \"1dcfdab9-27df-483f-a009-42303866394b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:10:00.698917174Z\",\n \"endTime\": \"2026-03-18T22:12:23.691383966Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a8e680>{ msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-fncbll\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\",\n \"name\": \"1dcfdab9-27df-483f-a009-42303866394b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:10:00.698917174Z\",\n \"endTime\": \"2026-03-18T22:12:23.691383966Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00094b230>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["503"], "Expires": ["-1"], "X-Ms-Request-Id": [ "ee86427e-005d-4cc8-bd01-2b82a0056536", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Content-Type-Options": ["nosniff"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Routing-Request-Id": [ "CANADAEAST:20260318T221233Z:a23b9d02-f439-4d75-bcef-bec5f70b2688", ], "X-Msedge-Ref": [ "Ref A: 2529055C24EC4D90ABA73CA7E8A5F278 Ref B: CHI261140504054 Ref C: 2026-03-18T22:12:33Z", ], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Envoy-Upstream-Service-Time": ["11"], "X-Ms-Correlation-Request-Id": [ "a23b9d02-f439-4d75-bcef-bec5f70b2688", ], "Strict-Transport-Security": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error: <fmt.wrapError \| 0xc000a8e6a0>: failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-fncbll" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b", "name": "1dcfdab9-27df-483f-a009-42303866394b", "status": "Failed", "startTime": "2026-03-18T22:10:00.698917174Z", "endTime": "2026-03-18T22:12:23.691383966Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-fncbll\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\",\n \"name\": \"1dcfdab9-27df-483f-a009-42303866394b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:10:00.698917174Z\",\n \"endTime\": \"2026-03-18T22:12:23.691383966Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000a8e680>{ msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-fncbll\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\",\n \"name\": \"1dcfdab9-27df-483f-a009-42303866394b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:10:00.698917174Z\",\n \"endTime\": \"2026-03-18T22:12:23.691383966Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00094b230>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["503"], "Expires": ["-1"], "X-Ms-Request-Id": [ "ee86427e-005d-4cc8-bd01-2b82a0056536", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Content-Type-Options": ["nosniff"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Routing-Request-Id": [ "CANADAEAST:20260318T221233Z:a23b9d02-f439-4d75-bcef-bec5f70b2688", ], "X-Msedge-Ref": [ "Ref A: 2529055C24EC4D90ABA73CA7E8A5F278 Ref B: CHI261140504054 Ref C: 2026-03-18T22:12:33Z", ], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Envoy-Upstream-Service-Time": ["11"], "X-Ms-Correlation-Request-Id": [ "a23b9d02-f439-4d75-bcef-bec5f70b2688", ], "Strict-Transport-Security": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 21:39:24 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc001368740>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-5g465c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8", "name": "90443756-21e0-463a-9d79-af79062353c8", "status": "Failed", "startTime": "2026-03-18T22:03:11.273093527Z", "endTime": "2026-03-18T22:04:50.652269195Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5g465c\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8\",\n \"name\": \"90443756-21e0-463a-9d79-af79062353c8\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:03:11.273093527Z\",\n \"endTime\": \"2026-03-18T22:04:50.652269195Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001368720>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5g465c\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8\",\n \"name\": \"90443756-21e0-463a-9d79-af79062353c8\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:03:11.273093527Z\",\n \"endTime\": \"2026-03-18T22:04:50.652269195Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001024b10>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "Expires": ["-1"], "X-Ms-Request-Id": [ "322ecb3a-cca7-41b3-b384-68f9e35de431", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"], "X-Ms-Routing-Request-Id": [ "CANADAEAST:20260318T220453Z:8c67fafb-e096-4b92-8c0f-a92b31e7fba0", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: F9AD2BF3996148F2860FAE04B9F244ED Ref B: CHI261140504023 Ref C: 2026-03-18T22:04:53Z", ], "Content-Length": ["503"], "X-Envoy-Upstream-Service-Time": ["11"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"], "X-Ms-Correlation-Request-Id": [ "8c67fafb-e096-4b92-8c0f-a92b31e7fba0", ], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc001368740>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-5g465c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8", "name": "90443756-21e0-463a-9d79-af79062353c8", "status": "Failed", "startTime": "2026-03-18T22:03:11.273093527Z", "endTime": "2026-03-18T22:04:50.652269195Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5g465c\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8\",\n \"name\": \"90443756-21e0-463a-9d79-af79062353c8\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:03:11.273093527Z\",\n \"endTime\": \"2026-03-18T22:04:50.652269195Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001368720>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5g465c\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8\",\n \"name\": \"90443756-21e0-463a-9d79-af79062353c8\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:03:11.273093527Z\",\n \"endTime\": \"2026-03-18T22:04:50.652269195Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001024b10>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "Expires": ["-1"], "X-Ms-Request-Id": [ "322ecb3a-cca7-41b3-b384-68f9e35de431", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"], "X-Ms-Routing-Request-Id": [ "CANADAEAST:20260318T220453Z:8c67fafb-e096-4b92-8c0f-a92b31e7fba0", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: F9AD2BF3996148F2860FAE04B9F244ED Ref B: CHI261140504023 Ref C: 2026-03-18T22:04:53Z", ], "Content-Length": ["503"], "X-Envoy-Upstream-Service-Time": ["11"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"], "X-Ms-Correlation-Request-Id": [ "8c67fafb-e096-4b92-8c0f-a92b31e7fba0", ], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred Cluster should become ready within 45 minutes 21:39:24 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s. Cluster should become ready within 45 minutes Expected <bool>: false to be true fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s. Cluster should become ready within 45 minutes Expected <bool>: false to be true ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 21:39:24 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error: <fmt.wrapError \| 0xc000b943a0>: failed to create HCP cluster ea-list8n6lb4: failed waiting for cluster="ea-list8n6lb4" in resourcegroup="ea-list-gwfwm5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130", "name": "b5aa74e8-43d5-4cff-b520-63b6b9163130", "status": "Failed", "startTime": "2026-03-18T21:58:12.305820102Z", "endTime": "2026-03-18T21:59:35.853803496Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-list8n6lb4: failed waiting for cluster=\"ea-list8n6lb4\" in resourcegroup=\"ea-list-gwfwm5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n \"name\": \"b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:12.305820102Z\",\n \"endTime\": \"2026-03-18T21:59:35.853803496Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b94380>{ msg: "failed waiting for cluster=\"ea-list8n6lb4\" in resourcegroup=\"ea-list-gwfwm5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n \"name\": \"b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:12.305820102Z\",\n \"endTime\": \"2026-03-18T21:59:35.853803496Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0011866c0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260318T215943Z:3c196c67-a4c1-40eb-abc4-c5d8f5d91317", ], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Wed, 18 Mar 2026 21:59:43 GMT", ], "Pragma": ["no-cache"], "X-Ms-Request-Id": [ "e2863777-8b30-4013-8910-cf48bfc93c57", ], "X-Envoy-Upstream-Service-Time": ["11"], "X-Msedge-Ref": [ "Ref A: 694EB5D5CF094E458C2CC1B85768A697 Ref B: CHI261140510060 Ref C: 2026-03-18T21:59:43Z", ], "Content-Length": ["503"], "Content-Type": ["application/json"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Content-Type-Options": ["nosniff"], "Cache-Control": ["no-cache"], "Expires... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error: <fmt.wrapError \| 0xc000b943a0>: failed to create HCP cluster ea-list8n6lb4: failed waiting for cluster="ea-list8n6lb4" in resourcegroup="ea-list-gwfwm5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130", "name": "b5aa74e8-43d5-4cff-b520-63b6b9163130", "status": "Failed", "startTime": "2026-03-18T21:58:12.305820102Z", "endTime": "2026-03-18T21:59:35.853803496Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-list8n6lb4: failed waiting for cluster=\"ea-list8n6lb4\" in resourcegroup=\"ea-list-gwfwm5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n \"name\": \"b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:12.305820102Z\",\n \"endTime\": \"2026-03-18T21:59:35.853803496Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000b94380>{ msg: "failed waiting for cluster=\"ea-list8n6lb4\" in resourcegroup=\"ea-list-gwfwm5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n \"name\": \"b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:12.305820102Z\",\n \"endTime\": \"2026-03-18T21:59:35.853803496Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0011866c0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260318T215943Z:3c196c67-a4c1-40eb-abc4-c5d8f5d91317", ], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Wed, 18 Mar 2026 21:59:43 GMT", ], "Pragma": ["no-cache"], "X-Ms-Request-Id": [ "e2863777-8b30-4013-8910-cf48bfc93c57", ], "X-Envoy-Upstream-Service-Time": ["11"], "X-Msedge-Ref": [ "Ref A: 694EB5D5CF094E458C2CC1B85768A697 Ref B: CHI261140510060 Ref C: 2026-03-18T21:59:43Z", ], "Content-Length": ["503"], "Content-Type": ["application/json"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Content-Type-Options": ["nosniff"], "Cache-Control": ["no-cache"], "Expires... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 21:39:24 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error: <fmt.wrapError \| 0xc0000d9d40>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-6fzwzp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88", "name": "ac78d6e9-5a49-4cfb-a070-1206176f3f88", "status": "Failed", "startTime": "2026-03-18T22:03:11.889861811Z", "endTime": "2026-03-18T22:04:50.608355115Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-6fzwzp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88\",\n \"name\": \"ac78d6e9-5a49-4cfb-a070-1206176f3f88\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:03:11.889861811Z\",\n \"endTime\": \"2026-03-18T22:04:50.608355115Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0000d9d20>{ msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-6fzwzp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88\",\n \"name\": \"ac78d6e9-5a49-4cfb-a070-1206176f3f88\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:03:11.889861811Z\",\n \"endTime\": \"2026-03-18T22:04:50.608355115Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00168adb0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "3249ef9a-7b0f-4f55-b29c-298f2d17ac8a", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Date": [ "Wed, 18 Mar 2026 22:04:53 GMT", ], "Pragma": ["no-cache"], "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Correlation-Request-Id": [ "b917974a-e14f-4131-aa5f-6e2f9f5f3176", ], "Cache-Control": ["no-cache"], "Content-Length": ["503"], "Expires": ["-1"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "Content-Type": ["application/json"], "X-Ms-Operation-Identifier": [ "appId=c94dda6b-80b4-4144-a102-590bf0... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error: <fmt.wrapError \| 0xc0000d9d40>: failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-6fzwzp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88", "name": "ac78d6e9-5a49-4cfb-a070-1206176f3f88", "status": "Failed", "startTime": "2026-03-18T22:03:11.889861811Z", "endTime": "2026-03-18T22:04:50.608355115Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-6fzwzp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88\",\n \"name\": \"ac78d6e9-5a49-4cfb-a070-1206176f3f88\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:03:11.889861811Z\",\n \"endTime\": \"2026-03-18T22:04:50.608355115Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0000d9d20>{ msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-6fzwzp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88\",\n \"name\": \"ac78d6e9-5a49-4cfb-a070-1206176f3f88\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:03:11.889861811Z\",\n \"endTime\": \"2026-03-18T22:04:50.608355115Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00168adb0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Request-Id": [ "3249ef9a-7b0f-4f55-b29c-298f2d17ac8a", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Date": [ "Wed, 18 Mar 2026 22:04:53 GMT", ], "Pragma": ["no-cache"], "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Correlation-Request-Id": [ "b917974a-e14f-4131-aa5f-6e2f9f5f3176", ], "Cache-Control": ["no-cache"], "Content-Length": ["503"], "Expires": ["-1"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "Content-Type": ["application/json"], "X-Ms-Operation-Identifier": [ "appId=c94dda6b-80b4-4144-a102-590bf0... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 21:39:24 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: Unexpected error: <fmt.wrapError \| 0xc0010a0360>: failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-xm72pt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e", "name": "0f5cf2e2-b390-4933-a78f-5efb244ef76e", "status": "Failed", "startTime": "2026-03-18T21:58:05.508161468Z", "endTime": "2026-03-18T21:59:35.825704307Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-xm72pt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e\",\n \"name\": \"0f5cf2e2-b390-4933-a78f-5efb244ef76e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:05.508161468Z\",\n \"endTime\": \"2026-03-18T21:59:35.825704307Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0010a0340>{ msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-xm72pt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e\",\n \"name\": \"0f5cf2e2-b390-4933-a78f-5efb244ef76e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:05.508161468Z\",\n \"endTime\": \"2026-03-18T21:59:35.825704307Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000ed06c0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Cache": ["CONFIG_NOCACHE"], "Content-Type": ["application/json"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/9e91e3e1-c55a-41c4-9fda-c6e81031aa4e", ], "X-Ms-Correlation-Request-Id": [ "7892e6be-859b-443d-93ae-25f63ec15cc7", ], "Cache-Control": ["no-cache"], "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Msedge-Ref": [ "Ref A: CBAC33FB359A45D5A9FC233D52AA531E Ref B: CHI261140506040 Ref C: 2026-03-18T21:59:37Z", ], "Date": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: Unexpected error: <fmt.wrapError \| 0xc0010a0360>: failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-xm72pt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e", "name": "0f5cf2e2-b390-4933-a78f-5efb244ef76e", "status": "Failed", "startTime": "2026-03-18T21:58:05.508161468Z", "endTime": "2026-03-18T21:59:35.825704307Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-xm72pt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e\",\n \"name\": \"0f5cf2e2-b390-4933-a78f-5efb244ef76e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:05.508161468Z\",\n \"endTime\": \"2026-03-18T21:59:35.825704307Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0010a0340>{ msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-xm72pt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e\",\n \"name\": \"0f5cf2e2-b390-4933-a78f-5efb244ef76e\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T21:58:05.508161468Z\",\n \"endTime\": \"2026-03-18T21:59:35.825704307Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000ed06c0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Cache": ["CONFIG_NOCACHE"], "Content-Type": ["application/json"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/9e91e3e1-c55a-41c4-9fda-c6e81031aa4e", ], "X-Ms-Correlation-Request-Id": [ "7892e6be-859b-443d-93ae-25f63ec15cc7", ], "Cache-Control": ["no-cache"], "X-Envoy-Upstream-Service-Time": ["9"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Msedge-Ref": [ "Ref A: CBAC33FB359A45D5A9FC233D52AA531E Ref B: CHI261140506040 Ref C: 2026-03-18T21:59:37Z", ], "Date": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: MultipleErrorsOccurred; detail code InternalServerError; detail message ClientCertificateCredential: unable to resolve an endpoint: server response error: context deadline exceeded; provider Microsoft.RedHatOpenShift 21:39:24 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error: <fmt.wrapError \| 0xc001828480>: failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-99xb7j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b", "name": "6e36926c-e6c5-49f7-a607-9f56a4bbb73b", "status": "Failed", "startTime": "2026-03-18T22:05:01.130113225Z", "endTime": "2026-03-18T22:06:29.307118365Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" }, { "code": "InternalServerError", "message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" }, { "code": "InternalServerError", "message": "failed to get quota usage data: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" }, { "code": "InternalServerError", "message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" }, { "code": "InternalServerError", "message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster patch-name-cluster: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-99xb7j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n \"name\": \"6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:05:01.130113225Z\",\n \"endTime\": \"2026-03-18T22:06:29.307118365Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get quota usage data: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001828460>{ msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-99xb7j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n \"name\": \"6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:05:01.130113225Z\",\n \"endTime\": \"2026-03-18T22:06:29.307118365Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerErro... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error: <fmt.wrapError \| 0xc001828480>: failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-99xb7j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: MultipleErrorsOccurred -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b", "name": "6e36926c-e6c5-49f7-a607-9f56a4bbb73b", "status": "Failed", "startTime": "2026-03-18T22:05:01.130113225Z", "endTime": "2026-03-18T22:06:29.307118365Z", "error": { "code": "MultipleErrorsOccurred", "message": "Cluster provisioning failed due to multiple errors", "details": [ { "code": "InternalServerError", "message": "Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" }, { "code": "InternalServerError", "message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" }, { "code": "InternalServerError", "message": "failed to get quota usage data: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" }, { "code": "InternalServerError", "message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" }, { "code": "InternalServerError", "message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster patch-name-cluster: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-99xb7j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n \"name\": \"6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:05:01.130113225Z\",\n \"endTime\": \"2026-03-18T22:06:29.307118365Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"failed to get quota usage data: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001828460>{ msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-99xb7j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n \"name\": \"6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:05:01.130113225Z\",\n \"endTime\": \"2026-03-18T22:06:29.307118365Z\",\n \"error\": {\n \"code\": \"MultipleErrorsOccurred\",\n \"message\": \"Cluster provisioning failed due to multiple errors\",\n \"details\": [\n {\n \"code\": \"InternalServerError\",\n \"message\": \"Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerError\",\n \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n },\n {\n \"code\": \"InternalServerErro... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 21:39:24 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc0013d8460>: failed to create HCP cluster cluster-zstream-4-19-lnsqh6: failed waiting for cluster="cluster-zstream-4-19-lnsqh6" in resourcegroup="rg-zstream-upgrade-4-19-pld4fl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884", "name": "5ed3389f-0b4e-4794-8c26-5b3ae8de8884", "status": "Failed", "startTime": "2026-03-18T22:09:50.25277888Z", "endTime": "2026-03-18T22:11:58.43033621Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-lnsqh6: failed waiting for cluster=\"cluster-zstream-4-19-lnsqh6\" in resourcegroup=\"rg-zstream-upgrade-4-19-pld4fl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884\",\n \"name\": \"5ed3389f-0b4e-4794-8c26-5b3ae8de8884\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:09:50.25277888Z\",\n \"endTime\": \"2026-03-18T22:11:58.43033621Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0013d8440>{ msg: "failed waiting for cluster=\"cluster-zstream-4-19-lnsqh6\" in resourcegroup=\"rg-zstream-upgrade-4-19-pld4fl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884\",\n \"name\": \"5ed3389f-0b4e-4794-8c26-5b3ae8de8884\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:09:50.25277888Z\",\n \"endTime\": \"2026-03-18T22:11:58.43033621Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0010687e0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Expires": ["-1"], "X-Ms-Correlation-Request-Id": [ "748d029f-96fc-49eb-a4e6-e71ccf950083", ], "Cache-Control": ["no-cache"], "X-Ms-Request-Id": [ "54e97da7-c63e-4bd9-a811-45e592ebb83d", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: 898F5B667A1D4266917476DA83548256 Ref B: CHI261140509036 Ref C: 2026-03-18T22:12:02Z", ], "Pragma": ["no-cache"], "Content-Length": ["501"], "Content-Type": ["application/json"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/c4c65aed-8a31-40d7-afaa-2606c73c3774", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc0013d8460>: failed to create HCP cluster cluster-zstream-4-19-lnsqh6: failed waiting for cluster="cluster-zstream-4-19-lnsqh6" in resourcegroup="rg-zstream-upgrade-4-19-pld4fl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884", "name": "5ed3389f-0b4e-4794-8c26-5b3ae8de8884", "status": "Failed", "startTime": "2026-03-18T22:09:50.25277888Z", "endTime": "2026-03-18T22:11:58.43033621Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-lnsqh6: failed waiting for cluster=\"cluster-zstream-4-19-lnsqh6\" in resourcegroup=\"rg-zstream-upgrade-4-19-pld4fl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884\",\n \"name\": \"5ed3389f-0b4e-4794-8c26-5b3ae8de8884\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:09:50.25277888Z\",\n \"endTime\": \"2026-03-18T22:11:58.43033621Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0013d8440>{ msg: "failed waiting for cluster=\"cluster-zstream-4-19-lnsqh6\" in resourcegroup=\"rg-zstream-upgrade-4-19-pld4fl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884\",\n \"name\": \"5ed3389f-0b4e-4794-8c26-5b3ae8de8884\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:09:50.25277888Z\",\n \"endTime\": \"2026-03-18T22:11:58.43033621Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0010687e0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Expires": ["-1"], "X-Ms-Correlation-Request-Id": [ "748d029f-96fc-49eb-a4e6-e71ccf950083", ], "Cache-Control": ["no-cache"], "X-Ms-Request-Id": [ "54e97da7-c63e-4bd9-a811-45e592ebb83d", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Msedge-Ref": [ "Ref A: 898F5B667A1D4266917476DA83548256 Ref B: CHI261140509036 Ref C: 2026-03-18T22:12:02Z", ], "Pragma": ["no-cache"], "Content-Length": ["501"], "Content-Type": ["application/json"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/c4c65aed-8a31-40d7-afaa-2606c73c3774", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 21:39:24 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000962220>: failed to create HCP cluster cluster-zstream-4-20-hfk97l: failed waiting for cluster="cluster-zstream-4-20-hfk97l" in resourcegroup="rg-zstream-upgrade-4-20-6jjt78" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49", "name": "a535b9a7-79bf-4ae9-99b6-679319213e49", "status": "Failed", "startTime": "2026-03-18T22:10:00.515143027Z", "endTime": "2026-03-18T22:11:58.502121438Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-hfk97l: failed waiting for cluster=\"cluster-zstream-4-20-hfk97l\" in resourcegroup=\"rg-zstream-upgrade-4-20-6jjt78\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\",\n \"name\": \"a535b9a7-79bf-4ae9-99b6-679319213e49\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:10:00.515143027Z\",\n \"endTime\": \"2026-03-18T22:11:58.502121438Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0009621e0>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-hfk97l\" in resourcegroup=\"rg-zstream-upgrade-4-20-6jjt78\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\",\n \"name\": \"a535b9a7-79bf-4ae9-99b6-679319213e49\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:10:00.515143027Z\",\n \"endTime\": \"2026-03-18T22:11:58.502121438Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000e803c0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/a23e4d00-7c12-428e-b637-d6e87c428e63", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260318T221203Z:86a9a285-d268-4eaa-81f2-1b43b6865e4f", ], "Cache-Control": ["no-cache"], "Content-Length": ["503"], "X-Envoy-Upstream-Service-Time": ["10"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Correlation-Request-Id": [ "86a9a285-d268-4eaa-81f2-1b43b6865e4f", ], "X-Msedge-Ref": [ "Ref A: 30110B94D9A9488595249B68A2A891EF Ref B: CHI261140505062 Ref C: 2026-03-18T22:12:03Z", ], "Date": [ "Wed, 18 Mar 2026 22:12... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc000962220>: failed to create HCP cluster cluster-zstream-4-20-hfk97l: failed waiting for cluster="cluster-zstream-4-20-hfk97l" in resourcegroup="rg-zstream-upgrade-4-20-6jjt78" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49", "name": "a535b9a7-79bf-4ae9-99b6-679319213e49", "status": "Failed", "startTime": "2026-03-18T22:10:00.515143027Z", "endTime": "2026-03-18T22:11:58.502121438Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-20-hfk97l: failed waiting for cluster=\"cluster-zstream-4-20-hfk97l\" in resourcegroup=\"rg-zstream-upgrade-4-20-6jjt78\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\",\n \"name\": \"a535b9a7-79bf-4ae9-99b6-679319213e49\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:10:00.515143027Z\",\n \"endTime\": \"2026-03-18T22:11:58.502121438Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0009621e0>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-hfk97l\" in resourcegroup=\"rg-zstream-upgrade-4-20-6jjt78\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\",\n \"name\": \"a535b9a7-79bf-4ae9-99b6-679319213e49\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-18T22:10:00.515143027Z\",\n \"endTime\": \"2026-03-18T22:11:58.502121438Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000e803c0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/a23e4d00-7c12-428e-b637-d6e87c428e63", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260318T221203Z:86a9a285-d268-4eaa-81f2-1b43b6865e4f", ], "Cache-Control": ["no-cache"], "Content-Length": ["503"], "X-Envoy-Upstream-Service-Time": ["10"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Correlation-Request-Id": [ "86a9a285-d268-4eaa-81f2-1b43b6865e4f", ], "X-Msedge-Ref": [ "Ref A: 30110B94D9A9488595249B68A2A891EF Ref B: CHI261140505062 Ref C: 2026-03-18T22:12:03Z", ], "Date": [ "Wed, 18 Mar 2026 22:12... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
08:09:22	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
01:48:26	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
00:56:26	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
00:51:49	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
00:46:51	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
00:45:27	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a
00:43:07	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	n/a	passed	n/a	0	n/a

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

21:39:24 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error:
    <*fmt.wrapError | 0xc0002e4c80>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-bnf7qs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c",
      "name": "a7266eb7-02fe-4e25-8a6b-80bf2b07af6c",
      "status": "Failed",
      "startTime": "2026-03-18T22:04:06.780626426Z",
      "endTime": "2026-03-18T22:04:50.639636342Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-bnf7qs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\",\n  \"name\": \"a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:04:06.780626426Z\",\n  \"endTime\": \"2026-03-18T22:04:50.639636342Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0002e4c60>{
            msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-bnf7qs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\",\n  \"name\": \"a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:04:06.780626426Z\",\n  \"endTime\": \"2026-03-18T22:04:50.639636342Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc001556900>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "68337901-b202-4387-8b62-a5d5fa9a8a87",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/86f586e0-8021-49ae-8dab-791e7e4ddc69",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Correlation-Request-Id": [
                            "b2f55a6e-086e-4aea-93f6-026a3332b1ba",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: 340E61C37B254A58B9B02493846AE154 Ref B: CHI261140503054 Ref C: 2026-03-18T22:04:57Z",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Ms-Ratelimit-Rem...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error:
    <*fmt.wrapError | 0xc0002e4c80>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-bnf7qs" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c",
      "name": "a7266eb7-02fe-4e25-8a6b-80bf2b07af6c",
      "status": "Failed",
      "startTime": "2026-03-18T22:04:06.780626426Z",
      "endTime": "2026-03-18T22:04:50.639636342Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-bnf7qs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\",\n  \"name\": \"a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:04:06.780626426Z\",\n  \"endTime\": \"2026-03-18T22:04:50.639636342Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0002e4c60>{
            msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-bnf7qs\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\",\n  \"name\": \"a7266eb7-02fe-4e25-8a6b-80bf2b07af6c\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:04:06.780626426Z\",\n  \"endTime\": \"2026-03-18T22:04:50.639636342Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc001556900>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "68337901-b202-4387-8b62-a5d5fa9a8a87",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/86f586e0-8021-49ae-8dab-791e7e4ddc69",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Correlation-Request-Id": [
                            "b2f55a6e-086e-4aea-93f6-026a3332b1ba",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: 340E61C37B254A58B9B02493846AE154 Ref B: CHI261140503054 Ref C: 2026-03-18T22:04:57Z",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Ms-Ratelimit-Rem...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

21:39:24 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc00101e2d0>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nxd528" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nxd528\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001374460>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528",
            },
            <*fmt.wrapErrors | 0xc00101e2a0>{
                msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nxd528\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001374460>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc00101e2d0>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-nxd528" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nxd528\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001374460>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528",
            },
            <*fmt.wrapErrors | 0xc00101e2a0>{
                msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-nxd528\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001374460>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-nxd528",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

21:39:24 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e6cb10>: 
    failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-vm79bx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vm79bx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011648c0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx",
            },
            <*fmt.wrapErrors | 0xc000e6cae0>{
                msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vm79bx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0011648c0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e6cb10>: 
    failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-vm79bx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster basic-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vm79bx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011648c0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx",
            },
            <*fmt.wrapErrors | 0xc000e6cae0>{
                msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vm79bx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0011648c0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster basic-hcp-cluster in resource group basic-cluster-vm79bx",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

21:39:24 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapErrors | 0xc000fc8e70>: 
    failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-zdx4fc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-zdx4fc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011f5360>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc",
            },
            <*fmt.wrapErrors | 0xc000fc8e40>{
                msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-zdx4fc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0011f5360>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapErrors | 0xc000fc8e70>: 
    failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: failed waiting for cluster="hcp-cluster-np-128" in resourcegroup="clusternp128-zdx4fc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster hcp-cluster-np-128, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-zdx4fc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0011f5360>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc",
            },
            <*fmt.wrapErrors | 0xc000fc8e40>{
                msg: "failed waiting for cluster=\"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-zdx4fc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0011f5360>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster hcp-cluster-np-128 in resource group clusternp128-zdx4fc",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

Deserializaion Error: no output from command

21:39:24 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel

Show raw failure

> github.com/Azure/ARO-HCP/test/e2e.init.func15.2({0x7f54ec580e60, 0xc0009fb7d0})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126
                | 
                | By("waiting for first cluster to complete creation")
                > _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{
                | 	Frequency: framework.StandardPollInterval,
                | })
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x827b940?, 0xc0009fb7d0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 84
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 03/18/26 23:20:55.083
  "ts"="2026-03-18 23:20:55.083532" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-18 23:20:55.083598" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-kjjpjb" "resourceGroup"="customer-rg-2hmq5s"
  "ts"="2026-03-18 23:20:55.333961" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-kjjpjb" "resourceGroup"=""
  "ts"="2026-03-18 23:20:56.369508" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-wlxsb8" "resourceGroup"="customer-rg-2hmq5s"
  "ts"="2026-03-18 23:20:56.519635" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-wlxsb8" "resourceGroup"=""
  "ts"="2026-03-18 23:20:57.115820" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
  "ts"="2026-03-18 23:20:57.115955" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
  "ts"="2026-03-18 23:20:57.116022" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-18 23:20:57.117126" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-26"
  "ts"="2026-03-18 23:20:57.117146" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-110"
  "ts"="2026-03-18 23:20:57.117168" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-2hmq5s"
  "ts"="2026-03-18 23:20:57.117208" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-36"
  "ts"="2026-03-18 23:20:57.264846" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-110" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-18 23:20:57.327570" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-36" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-18 23:20:57.396893" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-110" "deployment"="msi-deployment-110"
  "ts"="2026-03-18 23:20:57.432057" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32"
  "ts"="2026-03-18 23:20:57.548409" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-36" "deployment"="msi-deployment-36"
  "ts"="2026-03-18 23:20:57.629848" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="customer-infra-basic-hcp-cluster2-wlxsb8"
  "ts"="2026-03-18 23:20:57.773291" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-26" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-18 23:20:57.819000" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
  "ts"="2026-03-18 23:20:57.904929" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="customer-infra-basic-hcp-cluster-kjjpjb"
  "ts"="2026-03-18 23:20:58.268158" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-26" "deployment"="msi-deployment-26"
  "ts"="2026-03-18 23:20:58.435912" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-18 23:20:58.436063" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-18 23:20:58.436192" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-2hmq5s"
Command Error: signal: killed
Deserializaion Error: no output from command
> github.com/Azure/ARO-HCP/test/e2e.init.func15.2({0x7f54ec580e60, 0xc0009fb7d0})
              /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:126
                | 
                | By("waiting for first cluster to complete creation")
                > _, err = poller1.PollUntilDone(ctx, &runtime.PollUntilDoneOptions{
                | 	Frequency: framework.StandardPollInterval,
                | })
            github.com/onsi/ginkgo/v2/internal.extractBodyFunction.func2({0x827b940?, 0xc0009fb7d0?})
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/node.go:465
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode.func3()
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:901
            github.com/onsi/ginkgo/v2/internal.(*Suite).runNode in goroutine 84
              /go/pkg/mod/github.com/openshift/onsi-ginkgo/v2@v2.6.1-0.20250416174521-4eb003743b54/internal/suite.go:888
    ------------------------------
    [FAILED] in [It] - /opt/app-root/src/github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:129 @ 03/18/26 23:20:55.083
  "ts"="2026-03-18 23:20:55.083532" "level"=0 "msg"="Commiting timing metadata."
  "ts"="2026-03-18 23:20:55.083598" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster-kjjpjb" "resourceGroup"="customer-rg-2hmq5s"
  "ts"="2026-03-18 23:20:55.333961" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster-kjjpjb" "resourceGroup"=""
  "ts"="2026-03-18 23:20:56.369508" "level"=0 "msg"="Dumping deployment operations." "deployment"="customer-infra-basic-hcp-cluster2-wlxsb8" "resourceGroup"="customer-rg-2hmq5s"
  "ts"="2026-03-18 23:20:56.519635" "level"=0 "msg"="Dumping deployment operations." "deployment"="mi-basic-hcp-cluster2-wlxsb8" "resourceGroup"=""
  "ts"="2026-03-18 23:20:57.115820" "level"=0 "msg"="Wrote timing metadata" "path"="/tmp/secret/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
  "ts"="2026-03-18 23:20:57.115955" "level"=0 "msg"="Wrote timing metadata" "path"="/logs/artifacts/test-timing/timing-metadata-b8ed2189eb62b4145bd17832fa47015d3fc1ad56caa60a743b61b98540ac41f3.yaml.gz"
  "ts"="2026-03-18 23:20:57.116022" "level"=0 "msg"="collecting debug info"
  "ts"="2026-03-18 23:20:57.117126" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-26"
  "ts"="2026-03-18 23:20:57.117146" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-110"
  "ts"="2026-03-18 23:20:57.117168" "level"=0 "msg"="collecting deployments" "resourceGroup"="customer-rg-2hmq5s"
  "ts"="2026-03-18 23:20:57.117208" "level"=0 "msg"="collecting deployments" "resourceGroup"="aro-hcp-test-msi-containers-prod-36"
  "ts"="2026-03-18 23:20:57.264846" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-110" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-18 23:20:57.327570" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-36" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-18 23:20:57.396893" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-110" "deployment"="msi-deployment-110"
  "ts"="2026-03-18 23:20:57.432057" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="pooledNonMsiScopedAssignments-mooiin3edme32"
  "ts"="2026-03-18 23:20:57.548409" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-36" "deployment"="msi-deployment-36"
  "ts"="2026-03-18 23:20:57.629848" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="customer-infra-basic-hcp-cluster2-wlxsb8"
  "ts"="2026-03-18 23:20:57.773291" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-26" "deployment"="pooledMsiScopedAssignments"
  "ts"="2026-03-18 23:20:57.819000" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="pooledNonMsiScopedAssignments-eqbgltuyzhpg4"
  "ts"="2026-03-18 23:20:57.904929" "level"=0 "msg"="collecting operations" "resourceGroup"="customer-rg-2hmq5s" "deployment"="customer-infra-basic-hcp-cluster-kjjpjb"
  "ts"="2026-03-18 23:20:58.268158" "level"=0 "msg"="collecting operations" "resourceGroup"="aro-hcp-test-msi-containers-prod-26" "deployment"="msi-deployment-26"
  "ts"="2026-03-18 23:20:58.435912" "level"=0 "msg"="finished collecting debug info"
  "ts"="2026-03-18 23:20:58.436063" "level"=0 "msg"="deleting created resources"
  "ts"="2026-03-18 23:20:58.436192" "level"=0 "msg"="deleting all hcp clusters in resource group" "resourceGroup"="customer-rg-2hmq5s"
Command Error: signal: killed
Deserializaion Error: no output from command

timeout during CreateHCPClusterAndWait; context deadline exceeded

21:39:24 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014de840>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-txz9bx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-txz9bx, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-txz9bx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-txz9bx, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000c651e0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-txz9bx",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014de840>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-txz9bx" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-txz9bx, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-txz9bx\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-txz9bx, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000c651e0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-txz9bx",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

21:39:24 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0009001a0>: 
    failed to create HCP cluster cluster-zstream-4-21-94f5n8: failed waiting for cluster="cluster-zstream-4-21-94f5n8" in resourcegroup="rg-zstream-upgrade-4-21-64j8h4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1",
      "name": "8cac1b2e-d936-402c-86be-44b7a37d4ae1",
      "status": "Failed",
      "startTime": "2026-03-18T22:09:58.306382688Z",
      "endTime": "2026-03-18T22:11:58.452014263Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-21-94f5n8: failed waiting for cluster=\"cluster-zstream-4-21-94f5n8\" in resourcegroup=\"rg-zstream-upgrade-4-21-64j8h4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1\",\n  \"name\": \"8cac1b2e-d936-402c-86be-44b7a37d4ae1\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:09:58.306382688Z\",\n  \"endTime\": \"2026-03-18T22:11:58.452014263Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000900180>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-21-94f5n8\" in resourcegroup=\"rg-zstream-upgrade-4-21-64j8h4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1\",\n  \"name\": \"8cac1b2e-d936-402c-86be-44b7a37d4ae1\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:09:58.306382688Z\",\n  \"endTime\": \"2026-03-18T22:11:58.452014263Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000575020>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260318T221200Z:cc127123-120b-4dba-b721-6d7a1221b757",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "88b27e98-71ad-423d-941a-3dfed915fe8e",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "cc127123-120b-4dba-b721-6d7a1221b757",
                        ],
                        "Date": [
                            "Wed, 18 Mar 2026 22:12:00 GMT",
                        ],
                        "Content-Length": ["503"],
                        "Expires": ["-1"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/f9182697-47ef-450e-985b-d1bab32508ff",
                        ],
      ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0009001a0>: 
    failed to create HCP cluster cluster-zstream-4-21-94f5n8: failed waiting for cluster="cluster-zstream-4-21-94f5n8" in resourcegroup="rg-zstream-upgrade-4-21-64j8h4" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1",
      "name": "8cac1b2e-d936-402c-86be-44b7a37d4ae1",
      "status": "Failed",
      "startTime": "2026-03-18T22:09:58.306382688Z",
      "endTime": "2026-03-18T22:11:58.452014263Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-21-94f5n8: failed waiting for cluster=\"cluster-zstream-4-21-94f5n8\" in resourcegroup=\"rg-zstream-upgrade-4-21-64j8h4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1\",\n  \"name\": \"8cac1b2e-d936-402c-86be-44b7a37d4ae1\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:09:58.306382688Z\",\n  \"endTime\": \"2026-03-18T22:11:58.452014263Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000900180>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-21-94f5n8\" in resourcegroup=\"rg-zstream-upgrade-4-21-64j8h4\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8cac1b2e-d936-402c-86be-44b7a37d4ae1\",\n  \"name\": \"8cac1b2e-d936-402c-86be-44b7a37d4ae1\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:09:58.306382688Z\",\n  \"endTime\": \"2026-03-18T22:11:58.452014263Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000575020>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260318T221200Z:cc127123-120b-4dba-b721-6d7a1221b757",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "88b27e98-71ad-423d-941a-3dfed915fe8e",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "cc127123-120b-4dba-b721-6d7a1221b757",
                        ],
                        "Date": [
                            "Wed, 18 Mar 2026 22:12:00 GMT",
                        ],
                        "Content-Length": ["503"],
                        "Expires": ["-1"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/f9182697-47ef-450e-985b-d1bab32508ff",
                        ],
      ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

21:39:24 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc000de6db0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-fjpggg" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-fjpggg\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001575960>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg",
            },
            <*fmt.wrapErrors | 0xc000de6d80>{
                msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-fjpggg\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001575960>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc000de6db0>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-fjpggg" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-fjpggg\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001575960>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg",
            },
            <*fmt.wrapErrors | 0xc000de6d80>{
                msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-fjpggg\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001575960>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-fjpggg",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

21:39:24 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
    <*fmt.wrapError | 0xc000a02180>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-dngc6m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306",
      "name": "e959e2ef-e55d-46f1-8f37-18a399818306",
      "status": "Failed",
      "startTime": "2026-03-18T21:58:25.256129849Z",
      "endTime": "2026-03-18T21:59:35.785955056Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-dngc6m\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306\",\n  \"name\": \"e959e2ef-e55d-46f1-8f37-18a399818306\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T21:58:25.256129849Z\",\n  \"endTime\": \"2026-03-18T21:59:35.785955056Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a02160>{
            msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-dngc6m\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306\",\n  \"name\": \"e959e2ef-e55d-46f1-8f37-18a399818306\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T21:58:25.256129849Z\",\n  \"endTime\": \"2026-03-18T21:59:35.785955056Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000978450>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Correlation-Request-Id": [
                            "6b4b037b-5eae-4b4a-89a0-993dc36429cf",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Date": [
                            "Wed, 18 Mar 2026 21:59:36 GMT",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/7240e2d5-207e-4f6d-805f-097d383fb77f",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADAEAST:20260318T215936Z:6b4b037b-5eae-4b4a-89a0-993dc36429cf",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Cache-Control": ["no-...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
    <*fmt.wrapError | 0xc000a02180>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-dngc6m" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306",
      "name": "e959e2ef-e55d-46f1-8f37-18a399818306",
      "status": "Failed",
      "startTime": "2026-03-18T21:58:25.256129849Z",
      "endTime": "2026-03-18T21:59:35.785955056Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-dngc6m\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306\",\n  \"name\": \"e959e2ef-e55d-46f1-8f37-18a399818306\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T21:58:25.256129849Z\",\n  \"endTime\": \"2026-03-18T21:59:35.785955056Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a02160>{
            msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-dngc6m\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/e959e2ef-e55d-46f1-8f37-18a399818306\",\n  \"name\": \"e959e2ef-e55d-46f1-8f37-18a399818306\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T21:58:25.256129849Z\",\n  \"endTime\": \"2026-03-18T21:59:35.785955056Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000978450>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Correlation-Request-Id": [
                            "6b4b037b-5eae-4b4a-89a0-993dc36429cf",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Date": [
                            "Wed, 18 Mar 2026 21:59:36 GMT",
                        ],
                        "Content-Type": ["application/json"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/7240e2d5-207e-4f6d-805f-097d383fb77f",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADAEAST:20260318T215936Z:6b4b037b-5eae-4b4a-89a0-993dc36429cf",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Cache-Control": ["no-...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

21:39:24 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
    <*fmt.wrapError | 0xc0009927a0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-2x5tqd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c",
      "name": "15b94b31-6297-4e0a-818d-5ea9d03e264c",
      "status": "Failed",
      "startTime": "2026-03-18T21:58:09.542609231Z",
      "endTime": "2026-03-18T21:59:35.888371839Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-2x5tqd\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c\",\n  \"name\": \"15b94b31-6297-4e0a-818d-5ea9d03e264c\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T21:58:09.542609231Z\",\n  \"endTime\": \"2026-03-18T21:59:35.888371839Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000992780>{
            msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-2x5tqd\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c\",\n  \"name\": \"15b94b31-6297-4e0a-818d-5ea9d03e264c\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T21:58:09.542609231Z\",\n  \"endTime\": \"2026-03-18T21:59:35.888371839Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000928b40>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["503"],
                        "X-Ms-Correlation-Request-Id": [
                            "85a78c11-e075-4480-8bc1-622a8ec78799",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: 5C0E582900C8470A8392AAB9D62C27B0 Ref B: CHI261140504040 Ref C: 2026-03-18T21:59:41Z",
                        ],
                        "Date": [
                            "Wed, 18 Mar 2026 21:59:40 GMT",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"],
                        "Pragma": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "ee2f9f9d-ea91-4be1-8440-27274248be3e",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:83]: Unexpected error:
    <*fmt.wrapError | 0xc0009927a0>: 
    failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster="np-autoscale-cluster" in resourcegroup="np-autoscaling-2x5tqd" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c",
      "name": "15b94b31-6297-4e0a-818d-5ea9d03e264c",
      "status": "Failed",
      "startTime": "2026-03-18T21:58:09.542609231Z",
      "endTime": "2026-03-18T21:59:35.888371839Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster np-autoscale-cluster: failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-2x5tqd\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c\",\n  \"name\": \"15b94b31-6297-4e0a-818d-5ea9d03e264c\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T21:58:09.542609231Z\",\n  \"endTime\": \"2026-03-18T21:59:35.888371839Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000992780>{
            msg: "failed waiting for cluster=\"np-autoscale-cluster\" in resourcegroup=\"np-autoscaling-2x5tqd\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/15b94b31-6297-4e0a-818d-5ea9d03e264c\",\n  \"name\": \"15b94b31-6297-4e0a-818d-5ea9d03e264c\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T21:58:09.542609231Z\",\n  \"endTime\": \"2026-03-18T21:59:35.888371839Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000928b40>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["503"],
                        "X-Ms-Correlation-Request-Id": [
                            "85a78c11-e075-4480-8bc1-622a8ec78799",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: 5C0E582900C8470A8392AAB9D62C27B0 Ref B: CHI261140504040 Ref C: 2026-03-18T21:59:41Z",
                        ],
                        "Date": [
                            "Wed, 18 Mar 2026 21:59:40 GMT",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"],
                        "Pragma": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "ee2f9f9d-ea91-4be1-8440-27274248be3e",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

21:39:24 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc00142ccf0>: 
    failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-sjngpl" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-sjngpl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00158dc10>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl",
            },
            <*fmt.wrapErrors | 0xc00142ccc0>{
                msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-sjngpl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00158dc10>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc00142ccf0>: 
    failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-sjngpl" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-sjngpl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc00158dc10>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl",
            },
            <*fmt.wrapErrors | 0xc00142ccc0>{
                msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-sjngpl\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc00158dc10>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-sjngpl",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

ERROR CODE: MultipleErrorsOccurred; detail code InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

21:39:24 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapError | 0xc00133a3c0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-sv2zcp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e",
      "name": "b06360bf-0a54-4e7d-b9f1-3e5b6626253e",
      "status": "Failed",
      "startTime": "2026-03-18T22:17:00.823479898Z",
      "endTime": "2026-03-18T22:18:09.152905722Z",
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "Cluster provisioning failed due to multiple errors",
        "details": [
          {
            "code": "InternalServerError",
            "message": "insufficient public IP address quota: required 2, available 0"
          },
          {
            "code": "InternalServerError",
            "message": "Failed to get virtual network 'customer-vnet-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          },
          {
            "code": "InternalServerError",
            "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-120/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          },
          {
            "code": "InternalServerError",
            "message": "failed to get network security group 'customer-nsg-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          },
          {
            "code": "InternalServerError",
            "message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-sv2zcp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n  \"name\": \"b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:17:00.823479898Z\",\n  \"endTime\": \"2026-03-18T22:18:09.152905722Z\",\n  \"error\": {\n    \"code\": \"MultipleErrorsOccurred\",\n    \"message\": \"Cluster provisioning failed due to multiple errors\",\n    \"details\": [\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"insufficient public IP address quota: required 2, available 0\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"Failed to get virtual network 'customer-vnet-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-120/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"failed to get network security group 'customer-nsg-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00133a3a0>{
            msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-sv2zcp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n  \"name\": \"b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:17:00.823479898Z\",\n  \"endTime\": \"2026-03-18T22:18:09.152905722Z\",\n  \"error\": {\n    \"code\": \"MultipleErrorsOccurred\",\n    \"message\": \"Cluster provisioning failed due to multiple errors\",\n    \"details\": [\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"insufficient public IP address quota: required 2, available 0\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"Failed to get virtual network 'customer-vnet-name' details: Cl...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapError | 0xc00133a3c0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-sv2zcp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e",
      "name": "b06360bf-0a54-4e7d-b9f1-3e5b6626253e",
      "status": "Failed",
      "startTime": "2026-03-18T22:17:00.823479898Z",
      "endTime": "2026-03-18T22:18:09.152905722Z",
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "Cluster provisioning failed due to multiple errors",
        "details": [
          {
            "code": "InternalServerError",
            "message": "insufficient public IP address quota: required 2, available 0"
          },
          {
            "code": "InternalServerError",
            "message": "Failed to get virtual network 'customer-vnet-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          },
          {
            "code": "InternalServerError",
            "message": "failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-120/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          },
          {
            "code": "InternalServerError",
            "message": "failed to get network security group 'customer-nsg-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          },
          {
            "code": "InternalServerError",
            "message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-sv2zcp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n  \"name\": \"b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:17:00.823479898Z\",\n  \"endTime\": \"2026-03-18T22:18:09.152905722Z\",\n  \"error\": {\n    \"code\": \"MultipleErrorsOccurred\",\n    \"message\": \"Cluster provisioning failed due to multiple errors\",\n    \"details\": [\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"insufficient public IP address quota: required 2, available 0\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"Failed to get virtual network 'customer-vnet-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"failed to get managed identity '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/aro-hcp-test-msi-containers-prod-120/providers/Microsoft.ManagedIdentity/userAssignedIdentities/ingress': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"failed to get network security group 'customer-nsg-name' details: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00133a3a0>{
            msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-sv2zcp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n  \"name\": \"b06360bf-0a54-4e7d-b9f1-3e5b6626253e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:17:00.823479898Z\",\n  \"endTime\": \"2026-03-18T22:18:09.152905722Z\",\n  \"error\": {\n    \"code\": \"MultipleErrorsOccurred\",\n    \"message\": \"Cluster provisioning failed due to multiple errors\",\n    \"details\": [\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"insufficient public IP address quota: required 2, available 0\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"Failed to get virtual network 'customer-vnet-name' details: Cl...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

21:39:24 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010cc570>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-jrpfvc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-jrpfvc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fde070>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc",
            },
            <*fmt.wrapErrors | 0xc0010cc540>{
                msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-jrpfvc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000fde070>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapErrors | 0xc0010cc570>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-jrpfvc" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-jrpfvc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000fde070>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc",
            },
            <*fmt.wrapErrors | 0xc0010cc540>{
                msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-jrpfvc\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000fde070>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-jrpfvc",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

21:39:24 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapError | 0xc000a8e6a0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-fncbll" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b",
      "name": "1dcfdab9-27df-483f-a009-42303866394b",
      "status": "Failed",
      "startTime": "2026-03-18T22:10:00.698917174Z",
      "endTime": "2026-03-18T22:12:23.691383966Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-fncbll\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\",\n  \"name\": \"1dcfdab9-27df-483f-a009-42303866394b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:10:00.698917174Z\",\n  \"endTime\": \"2026-03-18T22:12:23.691383966Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a8e680>{
            msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-fncbll\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\",\n  \"name\": \"1dcfdab9-27df-483f-a009-42303866394b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:10:00.698917174Z\",\n  \"endTime\": \"2026-03-18T22:12:23.691383966Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00094b230>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["503"],
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "ee86427e-005d-4cc8-bd01-2b82a0056536",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADAEAST:20260318T221233Z:a23b9d02-f439-4d75-bcef-bec5f70b2688",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: 2529055C24EC4D90ABA73CA7E8A5F278 Ref B: CHI261140504054 Ref C: 2026-03-18T22:12:33Z",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Envoy-Upstream-Service-Time": ["11"],
                        "X-Ms-Correlation-Request-Id": [
                            "a23b9d02-f439-4d75-bcef-bec5f70b2688",
                        ],
                        "Strict-Transport-Security": [
     ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:93]: Unexpected error:
    <*fmt.wrapError | 0xc000a8e6a0>: 
    failed to create HCP cluster ea-cluster: failed waiting for cluster="ea-cluster" in resourcegroup="external-auth-cluster-fncbll" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b",
      "name": "1dcfdab9-27df-483f-a009-42303866394b",
      "status": "Failed",
      "startTime": "2026-03-18T22:10:00.698917174Z",
      "endTime": "2026-03-18T22:12:23.691383966Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ea-cluster: failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-fncbll\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\",\n  \"name\": \"1dcfdab9-27df-483f-a009-42303866394b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:10:00.698917174Z\",\n  \"endTime\": \"2026-03-18T22:12:23.691383966Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000a8e680>{
            msg: "failed waiting for cluster=\"ea-cluster\" in resourcegroup=\"external-auth-cluster-fncbll\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/1dcfdab9-27df-483f-a009-42303866394b\",\n  \"name\": \"1dcfdab9-27df-483f-a009-42303866394b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:10:00.698917174Z\",\n  \"endTime\": \"2026-03-18T22:12:23.691383966Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00094b230>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["503"],
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "ee86427e-005d-4cc8-bd01-2b82a0056536",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADAEAST:20260318T221233Z:a23b9d02-f439-4d75-bcef-bec5f70b2688",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: 2529055C24EC4D90ABA73CA7E8A5F278 Ref B: CHI261140504054 Ref C: 2026-03-18T22:12:33Z",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Envoy-Upstream-Service-Time": ["11"],
                        "X-Ms-Correlation-Request-Id": [
                            "a23b9d02-f439-4d75-bcef-bec5f70b2688",
                        ],
                        "Strict-Transport-Security": [
     ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

21:39:24 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapError | 0xc001368740>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-5g465c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8",
      "name": "90443756-21e0-463a-9d79-af79062353c8",
      "status": "Failed",
      "startTime": "2026-03-18T22:03:11.273093527Z",
      "endTime": "2026-03-18T22:04:50.652269195Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5g465c\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8\",\n  \"name\": \"90443756-21e0-463a-9d79-af79062353c8\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:03:11.273093527Z\",\n  \"endTime\": \"2026-03-18T22:04:50.652269195Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc001368720>{
            msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5g465c\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8\",\n  \"name\": \"90443756-21e0-463a-9d79-af79062353c8\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:03:11.273093527Z\",\n  \"endTime\": \"2026-03-18T22:04:50.652269195Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc001024b10>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Type": ["application/json"],
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "322ecb3a-cca7-41b3-b384-68f9e35de431",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADAEAST:20260318T220453Z:8c67fafb-e096-4b92-8c0f-a92b31e7fba0",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Msedge-Ref": [
                            "Ref A: F9AD2BF3996148F2860FAE04B9F244ED Ref B: CHI261140504023 Ref C: 2026-03-18T22:04:53Z",
                        ],
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["11"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"],
                        "X-Ms-Correlation-Request-Id": [
                            "8c67fafb-e096-4b92-8c0f-a92b31e7fba0",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Date": [
...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapError | 0xc001368740>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-5g465c" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8",
      "name": "90443756-21e0-463a-9d79-af79062353c8",
      "status": "Failed",
      "startTime": "2026-03-18T22:03:11.273093527Z",
      "endTime": "2026-03-18T22:04:50.652269195Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5g465c\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8\",\n  \"name\": \"90443756-21e0-463a-9d79-af79062353c8\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:03:11.273093527Z\",\n  \"endTime\": \"2026-03-18T22:04:50.652269195Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc001368720>{
            msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-5g465c\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/90443756-21e0-463a-9d79-af79062353c8\",\n  \"name\": \"90443756-21e0-463a-9d79-af79062353c8\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:03:11.273093527Z\",\n  \"endTime\": \"2026-03-18T22:04:50.652269195Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc001024b10>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Type": ["application/json"],
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "322ecb3a-cca7-41b3-b384-68f9e35de431",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3748"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADAEAST:20260318T220453Z:8c67fafb-e096-4b92-8c0f-a92b31e7fba0",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Msedge-Ref": [
                            "Ref A: F9AD2BF3996148F2860FAE04B9F244ED Ref B: CHI261140504023 Ref C: 2026-03-18T22:04:53Z",
                        ],
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["11"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["248"],
                        "X-Ms-Correlation-Request-Id": [
                            "8c67fafb-e096-4b92-8c0f-a92b31e7fba0",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Date": [
...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Cluster should become ready within 45 minutes

21:39:24 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:158]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

21:39:24 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapError | 0xc000b943a0>: 
    failed to create HCP cluster ea-list8n6lb4: failed waiting for cluster="ea-list8n6lb4" in resourcegroup="ea-list-gwfwm5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130",
      "name": "b5aa74e8-43d5-4cff-b520-63b6b9163130",
      "status": "Failed",
      "startTime": "2026-03-18T21:58:12.305820102Z",
      "endTime": "2026-03-18T21:59:35.853803496Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ea-list8n6lb4: failed waiting for cluster=\"ea-list8n6lb4\" in resourcegroup=\"ea-list-gwfwm5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n  \"name\": \"b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T21:58:12.305820102Z\",\n  \"endTime\": \"2026-03-18T21:59:35.853803496Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000b94380>{
            msg: "failed waiting for cluster=\"ea-list8n6lb4\" in resourcegroup=\"ea-list-gwfwm5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n  \"name\": \"b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T21:58:12.305820102Z\",\n  \"endTime\": \"2026-03-18T21:59:35.853803496Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0011866c0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260318T215943Z:3c196c67-a4c1-40eb-abc4-c5d8f5d91317",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Date": [
                            "Wed, 18 Mar 2026 21:59:43 GMT",
                        ],
                        "Pragma": ["no-cache"],
                        "X-Ms-Request-Id": [
                            "e2863777-8b30-4013-8910-cf48bfc93c57",
                        ],
                        "X-Envoy-Upstream-Service-Time": ["11"],
                        "X-Msedge-Ref": [
                            "Ref A: 694EB5D5CF094E458C2CC1B85768A697 Ref B: CHI261140510060 Ref C: 2026-03-18T21:59:43Z",
                        ],
                        "Content-Length": ["503"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Content-Type-Options": ["nosniff"],
                        "Cache-Control": ["no-cache"],
                        "Expires...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapError | 0xc000b943a0>: 
    failed to create HCP cluster ea-list8n6lb4: failed waiting for cluster="ea-list8n6lb4" in resourcegroup="ea-list-gwfwm5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130",
      "name": "b5aa74e8-43d5-4cff-b520-63b6b9163130",
      "status": "Failed",
      "startTime": "2026-03-18T21:58:12.305820102Z",
      "endTime": "2026-03-18T21:59:35.853803496Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ea-list8n6lb4: failed waiting for cluster=\"ea-list8n6lb4\" in resourcegroup=\"ea-list-gwfwm5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n  \"name\": \"b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T21:58:12.305820102Z\",\n  \"endTime\": \"2026-03-18T21:59:35.853803496Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000b94380>{
            msg: "failed waiting for cluster=\"ea-list8n6lb4\" in resourcegroup=\"ea-list-gwfwm5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n  \"name\": \"b5aa74e8-43d5-4cff-b520-63b6b9163130\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T21:58:12.305820102Z\",\n  \"endTime\": \"2026-03-18T21:59:35.853803496Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0011866c0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260318T215943Z:3c196c67-a4c1-40eb-abc4-c5d8f5d91317",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Date": [
                            "Wed, 18 Mar 2026 21:59:43 GMT",
                        ],
                        "Pragma": ["no-cache"],
                        "X-Ms-Request-Id": [
                            "e2863777-8b30-4013-8910-cf48bfc93c57",
                        ],
                        "X-Envoy-Upstream-Service-Time": ["11"],
                        "X-Msedge-Ref": [
                            "Ref A: 694EB5D5CF094E458C2CC1B85768A697 Ref B: CHI261140510060 Ref C: 2026-03-18T21:59:43Z",
                        ],
                        "Content-Length": ["503"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Content-Type-Options": ["nosniff"],
                        "Cache-Control": ["no-cache"],
                        "Expires...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

21:39:24 UTC · failed at e2e · test Customer should be able to update node pool labels and taints · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error:
    <*fmt.wrapError | 0xc0000d9d40>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-6fzwzp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88",
      "name": "ac78d6e9-5a49-4cfb-a070-1206176f3f88",
      "status": "Failed",
      "startTime": "2026-03-18T22:03:11.889861811Z",
      "endTime": "2026-03-18T22:04:50.608355115Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-6fzwzp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88\",\n  \"name\": \"ac78d6e9-5a49-4cfb-a070-1206176f3f88\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:03:11.889861811Z\",\n  \"endTime\": \"2026-03-18T22:04:50.608355115Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0000d9d20>{
            msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-6fzwzp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88\",\n  \"name\": \"ac78d6e9-5a49-4cfb-a070-1206176f3f88\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:03:11.889861811Z\",\n  \"endTime\": \"2026-03-18T22:04:50.608355115Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00168adb0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "3249ef9a-7b0f-4f55-b29c-298f2d17ac8a",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Date": [
                            "Wed, 18 Mar 2026 22:04:53 GMT",
                        ],
                        "Pragma": ["no-cache"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Correlation-Request-Id": [
                            "b917974a-e14f-4131-aa5f-6e2f9f5f3176",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["503"],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Operation-Identifier": [
                            "appId=c94dda6b-80b4-4144-a102-590bf0...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_labels_taints.go:91]: Unexpected error:
    <*fmt.wrapError | 0xc0000d9d40>: 
    failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster="cluster-np-labels-taints" in resourcegroup="rg-np-labels-taints-6fzwzp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88",
      "name": "ac78d6e9-5a49-4cfb-a070-1206176f3f88",
      "status": "Failed",
      "startTime": "2026-03-18T22:03:11.889861811Z",
      "endTime": "2026-03-18T22:04:50.608355115Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-np-labels-taints: failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-6fzwzp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88\",\n  \"name\": \"ac78d6e9-5a49-4cfb-a070-1206176f3f88\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:03:11.889861811Z\",\n  \"endTime\": \"2026-03-18T22:04:50.608355115Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0000d9d20>{
            msg: "failed waiting for cluster=\"cluster-np-labels-taints\" in resourcegroup=\"rg-np-labels-taints-6fzwzp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/ac78d6e9-5a49-4cfb-a070-1206176f3f88\",\n  \"name\": \"ac78d6e9-5a49-4cfb-a070-1206176f3f88\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:03:11.889861811Z\",\n  \"endTime\": \"2026-03-18T22:04:50.608355115Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00168adb0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Request-Id": [
                            "3249ef9a-7b0f-4f55-b29c-298f2d17ac8a",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Date": [
                            "Wed, 18 Mar 2026 22:04:53 GMT",
                        ],
                        "Pragma": ["no-cache"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Correlation-Request-Id": [
                            "b917974a-e14f-4131-aa5f-6e2f9f5f3176",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["503"],
                        "Expires": ["-1"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Operation-Identifier": [
                            "appId=c94dda6b-80b4-4144-a102-590bf0...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

21:39:24 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: Unexpected error:
    <*fmt.wrapError | 0xc0010a0360>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-xm72pt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e",
      "name": "0f5cf2e2-b390-4933-a78f-5efb244ef76e",
      "status": "Failed",
      "startTime": "2026-03-18T21:58:05.508161468Z",
      "endTime": "2026-03-18T21:59:35.825704307Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-xm72pt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e\",\n  \"name\": \"0f5cf2e2-b390-4933-a78f-5efb244ef76e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T21:58:05.508161468Z\",\n  \"endTime\": \"2026-03-18T21:59:35.825704307Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0010a0340>{
            msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-xm72pt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e\",\n  \"name\": \"0f5cf2e2-b390-4933-a78f-5efb244ef76e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T21:58:05.508161468Z\",\n  \"endTime\": \"2026-03-18T21:59:35.825704307Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000ed06c0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/9e91e3e1-c55a-41c4-9fda-c6e81031aa4e",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "7892e6be-859b-443d-93ae-25f63ec15cc7",
                        ],
                        "Cache-Control": ["no-cache"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: CBAC33FB359A45D5A9FC233D52AA531E Ref B: CHI261140506040 Ref C: 2026-03-18T21:59:37Z",
                        ],
                        "Date": [
                          ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:94]: Unexpected error:
    <*fmt.wrapError | 0xc0010a0360>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-xm72pt" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e",
      "name": "0f5cf2e2-b390-4933-a78f-5efb244ef76e",
      "status": "Failed",
      "startTime": "2026-03-18T21:58:05.508161468Z",
      "endTime": "2026-03-18T21:59:35.825704307Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster tls-endpoint-hcp-cluster: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-xm72pt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e\",\n  \"name\": \"0f5cf2e2-b390-4933-a78f-5efb244ef76e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T21:58:05.508161468Z\",\n  \"endTime\": \"2026-03-18T21:59:35.825704307Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0010a0340>{
            msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-xm72pt\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/0f5cf2e2-b390-4933-a78f-5efb244ef76e\",\n  \"name\": \"0f5cf2e2-b390-4933-a78f-5efb244ef76e\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T21:58:05.508161468Z\",\n  \"endTime\": \"2026-03-18T21:59:35.825704307Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000ed06c0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/9e91e3e1-c55a-41c4-9fda-c6e81031aa4e",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "7892e6be-859b-443d-93ae-25f63ec15cc7",
                        ],
                        "Cache-Control": ["no-cache"],
                        "X-Envoy-Upstream-Service-Time": ["9"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: CBAC33FB359A45D5A9FC233D52AA531E Ref B: CHI261140506040 Ref C: 2026-03-18T21:59:37Z",
                        ],
                        "Date": [
                          ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: MultipleErrorsOccurred; detail code InternalServerError; detail message ClientCertificateCredential: unable to resolve an endpoint: server response error: context deadline exceeded; provider Microsoft.RedHatOpenShift

21:39:24 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
    <*fmt.wrapError | 0xc001828480>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-99xb7j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b",
      "name": "6e36926c-e6c5-49f7-a607-9f56a4bbb73b",
      "status": "Failed",
      "startTime": "2026-03-18T22:05:01.130113225Z",
      "endTime": "2026-03-18T22:06:29.307118365Z",
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "Cluster provisioning failed due to multiple errors",
        "details": [
          {
            "code": "InternalServerError",
            "message": "Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          },
          {
            "code": "InternalServerError",
            "message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          },
          {
            "code": "InternalServerError",
            "message": "failed to get quota usage data: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          },
          {
            "code": "InternalServerError",
            "message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          },
          {
            "code": "InternalServerError",
            "message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster patch-name-cluster: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-99xb7j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n  \"name\": \"6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:05:01.130113225Z\",\n  \"endTime\": \"2026-03-18T22:06:29.307118365Z\",\n  \"error\": {\n    \"code\": \"MultipleErrorsOccurred\",\n    \"message\": \"Cluster provisioning failed due to multiple errors\",\n    \"details\": [\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"failed to get quota usage data: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc001828460>{
            msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-99xb7j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n  \"name\": \"6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:05:01.130113225Z\",\n  \"endTime\": \"2026-03-18T22:06:29.307118365Z\",\n  \"error\": {\n    \"code\": \"MultipleErrorsOccurred\",\n    \"message\": \"Cluster provisioning failed due to multiple errors\",\n    \"details\": [\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerErro...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
    <*fmt.wrapError | 0xc001828480>: 
    failed to create HCP cluster patch-name-cluster: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-99xb7j" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: MultipleErrorsOccurred
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b",
      "name": "6e36926c-e6c5-49f7-a607-9f56a4bbb73b",
      "status": "Failed",
      "startTime": "2026-03-18T22:05:01.130113225Z",
      "endTime": "2026-03-18T22:06:29.307118365Z",
      "error": {
        "code": "MultipleErrorsOccurred",
        "message": "Cluster provisioning failed due to multiple errors",
        "details": [
          {
            "code": "InternalServerError",
            "message": "Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          },
          {
            "code": "InternalServerError",
            "message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          },
          {
            "code": "InternalServerError",
            "message": "failed to get quota usage data: ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          },
          {
            "code": "InternalServerError",
            "message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          },
          {
            "code": "InternalServerError",
            "message": "ClientCertificateCredential: unable to resolve an endpoint: server response error:\n context deadline exceeded"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster patch-name-cluster: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-99xb7j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n  \"name\": \"6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:05:01.130113225Z\",\n  \"endTime\": \"2026-03-18T22:06:29.307118365Z\",\n  \"error\": {\n    \"code\": \"MultipleErrorsOccurred\",\n    \"message\": \"Cluster provisioning failed due to multiple errors\",\n    \"details\": [\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"failed to get quota usage data: ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc001828460>{
            msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-99xb7j\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: MultipleErrorsOccurred\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n  \"name\": \"6e36926c-e6c5-49f7-a607-9f56a4bbb73b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:05:01.130113225Z\",\n  \"endTime\": \"2026-03-18T22:06:29.307118365Z\",\n  \"error\": {\n    \"code\": \"MultipleErrorsOccurred\",\n    \"message\": \"Cluster provisioning failed due to multiple errors\",\n    \"details\": [\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"Failed to verify existence of managed resource group 'patch-name-99xb7j--managed': ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerError\",\n        \"message\": \"ClientCertificateCredential: unable to resolve an endpoint: server response error:\\n context deadline exceeded\"\n      },\n      {\n        \"code\": \"InternalServerErro...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

21:39:24 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0013d8460>: 
    failed to create HCP cluster cluster-zstream-4-19-lnsqh6: failed waiting for cluster="cluster-zstream-4-19-lnsqh6" in resourcegroup="rg-zstream-upgrade-4-19-pld4fl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884",
      "name": "5ed3389f-0b4e-4794-8c26-5b3ae8de8884",
      "status": "Failed",
      "startTime": "2026-03-18T22:09:50.25277888Z",
      "endTime": "2026-03-18T22:11:58.43033621Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-lnsqh6: failed waiting for cluster=\"cluster-zstream-4-19-lnsqh6\" in resourcegroup=\"rg-zstream-upgrade-4-19-pld4fl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884\",\n  \"name\": \"5ed3389f-0b4e-4794-8c26-5b3ae8de8884\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:09:50.25277888Z\",\n  \"endTime\": \"2026-03-18T22:11:58.43033621Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0013d8440>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-19-lnsqh6\" in resourcegroup=\"rg-zstream-upgrade-4-19-pld4fl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884\",\n  \"name\": \"5ed3389f-0b4e-4794-8c26-5b3ae8de8884\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:09:50.25277888Z\",\n  \"endTime\": \"2026-03-18T22:11:58.43033621Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0010687e0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Expires": ["-1"],
                        "X-Ms-Correlation-Request-Id": [
                            "748d029f-96fc-49eb-a4e6-e71ccf950083",
                        ],
                        "Cache-Control": ["no-cache"],
                        "X-Ms-Request-Id": [
                            "54e97da7-c63e-4bd9-a811-45e592ebb83d",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: 898F5B667A1D4266917476DA83548256 Ref B: CHI261140509036 Ref C: 2026-03-18T22:12:02Z",
                        ],
                        "Pragma": ["no-cache"],
                        "Content-Length": ["501"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/c4c65aed-8a31-40d7-afaa-2606c73c3774",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc0013d8460>: 
    failed to create HCP cluster cluster-zstream-4-19-lnsqh6: failed waiting for cluster="cluster-zstream-4-19-lnsqh6" in resourcegroup="rg-zstream-upgrade-4-19-pld4fl" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884",
      "name": "5ed3389f-0b4e-4794-8c26-5b3ae8de8884",
      "status": "Failed",
      "startTime": "2026-03-18T22:09:50.25277888Z",
      "endTime": "2026-03-18T22:11:58.43033621Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-lnsqh6: failed waiting for cluster=\"cluster-zstream-4-19-lnsqh6\" in resourcegroup=\"rg-zstream-upgrade-4-19-pld4fl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884\",\n  \"name\": \"5ed3389f-0b4e-4794-8c26-5b3ae8de8884\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:09:50.25277888Z\",\n  \"endTime\": \"2026-03-18T22:11:58.43033621Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0013d8440>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-19-lnsqh6\" in resourcegroup=\"rg-zstream-upgrade-4-19-pld4fl\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/5ed3389f-0b4e-4794-8c26-5b3ae8de8884\",\n  \"name\": \"5ed3389f-0b4e-4794-8c26-5b3ae8de8884\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:09:50.25277888Z\",\n  \"endTime\": \"2026-03-18T22:11:58.43033621Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0010687e0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Expires": ["-1"],
                        "X-Ms-Correlation-Request-Id": [
                            "748d029f-96fc-49eb-a4e6-e71ccf950083",
                        ],
                        "Cache-Control": ["no-cache"],
                        "X-Ms-Request-Id": [
                            "54e97da7-c63e-4bd9-a811-45e592ebb83d",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Msedge-Ref": [
                            "Ref A: 898F5B667A1D4266917476DA83548256 Ref B: CHI261140509036 Ref C: 2026-03-18T22:12:02Z",
                        ],
                        "Pragma": ["no-cache"],
                        "Content-Length": ["501"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadaeast/c4c65aed-8a31-40d7-afaa-2606c73c3774",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

21:39:24 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000962220>: 
    failed to create HCP cluster cluster-zstream-4-20-hfk97l: failed waiting for cluster="cluster-zstream-4-20-hfk97l" in resourcegroup="rg-zstream-upgrade-4-20-6jjt78" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49",
      "name": "a535b9a7-79bf-4ae9-99b6-679319213e49",
      "status": "Failed",
      "startTime": "2026-03-18T22:10:00.515143027Z",
      "endTime": "2026-03-18T22:11:58.502121438Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-20-hfk97l: failed waiting for cluster=\"cluster-zstream-4-20-hfk97l\" in resourcegroup=\"rg-zstream-upgrade-4-20-6jjt78\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\",\n  \"name\": \"a535b9a7-79bf-4ae9-99b6-679319213e49\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:10:00.515143027Z\",\n  \"endTime\": \"2026-03-18T22:11:58.502121438Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0009621e0>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-20-hfk97l\" in resourcegroup=\"rg-zstream-upgrade-4-20-6jjt78\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\",\n  \"name\": \"a535b9a7-79bf-4ae9-99b6-679319213e49\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:10:00.515143027Z\",\n  \"endTime\": \"2026-03-18T22:11:58.502121438Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000e803c0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/a23e4d00-7c12-428e-b637-d6e87c428e63",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260318T221203Z:86a9a285-d268-4eaa-81f2-1b43b6865e4f",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["10"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Correlation-Request-Id": [
                            "86a9a285-d268-4eaa-81f2-1b43b6865e4f",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: 30110B94D9A9488595249B68A2A891EF Ref B: CHI261140505062 Ref C: 2026-03-18T22:12:03Z",
                        ],
                        "Date": [
                            "Wed, 18 Mar 2026 22:12...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc000962220>: 
    failed to create HCP cluster cluster-zstream-4-20-hfk97l: failed waiting for cluster="cluster-zstream-4-20-hfk97l" in resourcegroup="rg-zstream-upgrade-4-20-6jjt78" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49",
      "name": "a535b9a7-79bf-4ae9-99b6-679319213e49",
      "status": "Failed",
      "startTime": "2026-03-18T22:10:00.515143027Z",
      "endTime": "2026-03-18T22:11:58.502121438Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-20-hfk97l: failed waiting for cluster=\"cluster-zstream-4-20-hfk97l\" in resourcegroup=\"rg-zstream-upgrade-4-20-6jjt78\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\",\n  \"name\": \"a535b9a7-79bf-4ae9-99b6-679319213e49\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:10:00.515143027Z\",\n  \"endTime\": \"2026-03-18T22:11:58.502121438Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0009621e0>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-20-hfk97l\" in resourcegroup=\"rg-zstream-upgrade-4-20-6jjt78\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/a535b9a7-79bf-4ae9-99b6-679319213e49\",\n  \"name\": \"a535b9a7-79bf-4ae9-99b6-679319213e49\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-18T22:10:00.515143027Z\",\n  \"endTime\": \"2026-03-18T22:11:58.502121438Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000e803c0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/a23e4d00-7c12-428e-b637-d6e87c428e63",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260318T221203Z:86a9a285-d268-4eaa-81f2-1b43b6865e4f",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["10"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Correlation-Request-Id": [
                            "86a9a285-d268-4eaa-81f2-1b43b6865e4f",
                        ],
                        "X-Msedge-Ref": [
                            "Ref A: 30110B94D9A9488595249B68A2A891EF Ref B: CHI261140505062 Ref C: 2026-03-18T22:12:03Z",
                        ],
                        "Date": [
                            "Wed, 18 Mar 2026 22:12...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Environment: PROD