CIHealth Run Log

↗ Open Failure patterns for this day

Environment: PROD

Time (UTC)	Job	Failed at	Result	PR	Failed tests	Details
05:06:22	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	22	e2e (22) timeout during CreateHCPClusterAndWait; context deadline exceeded 05:06:22 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error: <fmt.wrapErrors \| 0xc0014454d0>: failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-5lns4q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: context deadline exceeded { msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-5lns4q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001459930>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q", }, <fmt.wrapErrors \| 0xc0014454a0>{ msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-5lns4q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001459930>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error: <fmt.wrapErrors \| 0xc0014454d0>: failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-5lns4q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: context deadline exceeded { msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-5lns4q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001459930>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q", }, <fmt.wrapErrors \| 0xc0014454a0>{ msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-5lns4q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc001459930>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q", }, <context.deadlineExceededError>{}, ], }, ], } occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 05:06:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error: <fmt.wrapErrors \| 0xc00101cd20>: failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-lpp22g" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: context deadline exceeded { msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-lpp22g\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0008a2ae0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g", }, <fmt.wrapErrors \| 0xc00101ccf0>{ msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-lpp22g\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0008a2ae0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error: <fmt.wrapErrors \| 0xc00101cd20>: failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-lpp22g" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: context deadline exceeded { msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-lpp22g\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0008a2ae0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g", }, <fmt.wrapErrors \| 0xc00101ccf0>{ msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-lpp22g\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0008a2ae0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g", }, <context.deadlineExceededError>{}, ], }, ], } occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 05:06:22 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error: <fmt.wrapError \| 0xc0016292c0>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-6dmknb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487", "name": "d3e5222f-7626-4089-81ad-74ce5593b487", "status": "Failed", "startTime": "2026-03-16T05:53:52.774104349Z", "endTime": "2026-03-16T05:54:48.4817526Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-6dmknb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487\",\n \"name\": \"d3e5222f-7626-4089-81ad-74ce5593b487\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:53:52.774104349Z\",\n \"endTime\": \"2026-03-16T05:54:48.4817526Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0016292a0>{ msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-6dmknb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487\",\n \"name\": \"d3e5222f-7626-4089-81ad-74ce5593b487\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:53:52.774104349Z\",\n \"endTime\": \"2026-03-16T05:54:48.4817526Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001384930>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Content-Type-Options": ["nosniff"], "Date": [ "Mon, 16 Mar 2026 05:54:53 GMT", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/4a777810-db65-4717-9cdc-6501be453685", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260316T055453Z:b23331dc-7a87-4aac-85ec-6b1435706e75", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Cache-Control": ["no-cache"], "Content-Length": ["501"], "X-Envoy-Upstream-Service-Time": ["10"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error: <fmt.wrapError \| 0xc0016292c0>: failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-6dmknb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487", "name": "d3e5222f-7626-4089-81ad-74ce5593b487", "status": "Failed", "startTime": "2026-03-16T05:53:52.774104349Z", "endTime": "2026-03-16T05:54:48.4817526Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-6dmknb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487\",\n \"name\": \"d3e5222f-7626-4089-81ad-74ce5593b487\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:53:52.774104349Z\",\n \"endTime\": \"2026-03-16T05:54:48.4817526Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0016292a0>{ msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-6dmknb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487\",\n \"name\": \"d3e5222f-7626-4089-81ad-74ce5593b487\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:53:52.774104349Z\",\n \"endTime\": \"2026-03-16T05:54:48.4817526Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001384930>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Type": ["application/json"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Content-Type-Options": ["nosniff"], "Date": [ "Mon, 16 Mar 2026 05:54:53 GMT", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/4a777810-db65-4717-9cdc-6501be453685", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260316T055453Z:b23331dc-7a87-4aac-85ec-6b1435706e75", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "Cache-Control": ["no-cache"], "Content-Length": ["501"], "X-Envoy-Upstream-Service-Time": ["10"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 05:06:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error: <fmt.wrapError \| 0xc000695540>: failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-vgmwdp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2", "name": "8acc7e04-af91-4136-83d9-cfd4d620d5c2", "status": "Failed", "startTime": "2026-03-16T05:58:36.120361686Z", "endTime": "2026-03-16T05:59:25.865971075Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vgmwdp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2\",\n \"name\": \"8acc7e04-af91-4136-83d9-cfd4d620d5c2\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:58:36.120361686Z\",\n \"endTime\": \"2026-03-16T05:59:25.865971075Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000695500>{ msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vgmwdp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2\",\n \"name\": \"8acc7e04-af91-4136-83d9-cfd4d620d5c2\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:58:36.120361686Z\",\n \"endTime\": \"2026-03-16T05:59:25.865971075Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000adeed0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Expires": ["-1"], "X-Ms-Request-Id": [ "20c01661-0afe-48d2-8f91-e191ffbc139d", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260316T055926Z:6229d6a4-a060-4bcd-8adb-767c2b1c2524", ], "X-Content-Type-Options": ["nosniff"], "Content-Length": ["503"], "X-Envoy-Upstream-Service-Time": ["11"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/f93d9d56-0328-4fdd-b9c9-60f270da1dcd", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "Strict-Transport-Security": [ "max-age=3153... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error: <fmt.wrapError \| 0xc000695540>: failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-vgmwdp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2", "name": "8acc7e04-af91-4136-83d9-cfd4d620d5c2", "status": "Failed", "startTime": "2026-03-16T05:58:36.120361686Z", "endTime": "2026-03-16T05:59:25.865971075Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vgmwdp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2\",\n \"name\": \"8acc7e04-af91-4136-83d9-cfd4d620d5c2\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:58:36.120361686Z\",\n \"endTime\": \"2026-03-16T05:59:25.865971075Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000695500>{ msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vgmwdp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2\",\n \"name\": \"8acc7e04-af91-4136-83d9-cfd4d620d5c2\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:58:36.120361686Z\",\n \"endTime\": \"2026-03-16T05:59:25.865971075Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000adeed0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Expires": ["-1"], "X-Ms-Request-Id": [ "20c01661-0afe-48d2-8f91-e191ffbc139d", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260316T055926Z:6229d6a4-a060-4bcd-8adb-767c2b1c2524", ], "X-Content-Type-Options": ["nosniff"], "Content-Length": ["503"], "X-Envoy-Upstream-Service-Time": ["11"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/f93d9d56-0328-4fdd-b9c9-60f270da1dcd", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "Strict-Transport-Security": [ "max-age=3153... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 05:06:22 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapErrors \| 0xc00106e6f0>: failed to create HCP cluster cluster-zstream-4-21-jsjz55, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: failed waiting for cluster="cluster-zstream-4-21-jsjz55" in resourcegroup="rg-zstream-upgrade-4-21-b2bt9k" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: context deadline exceeded { msg: "failed to create HCP cluster cluster-zstream-4-21-jsjz55, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: failed waiting for cluster=\"cluster-zstream-4-21-jsjz55\" in resourcegroup=\"rg-zstream-upgrade-4-21-b2bt9k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000a91e10>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k", }, <fmt.wrapErrors \| 0xc00106e6c0>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-jsjz55\" in resourcegroup=\"rg-zstream-upgrade-4-21-b2bt9k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000a91e10>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapErrors \| 0xc00106e6f0>: failed to create HCP cluster cluster-zstream-4-21-jsjz55, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: failed waiting for cluster="cluster-zstream-4-21-jsjz55" in resourcegroup="rg-zstream-upgrade-4-21-b2bt9k" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: context deadline exceeded { msg: "failed to create HCP cluster cluster-zstream-4-21-jsjz55, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: failed waiting for cluster=\"cluster-zstream-4-21-jsjz55\" in resourcegroup=\"rg-zstream-upgrade-4-21-b2bt9k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000a91e10>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k", }, <fmt.wrapErrors \| 0xc00106e6c0>{ msg: "failed waiting for cluster=\"cluster-zstream-4-21-jsjz55\" in resourcegroup=\"rg-zstream-upgrade-4-21-b2bt9k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000a91e10>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k", }, <context.deadlineExceededError>{}, ], }, ], } occurred ERROR CODE: AuthorizationFailed; detail message The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '<uuid>' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.RedHatOpenShift 05:06:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error: <fmt.wrapError \| 0xc0008f2c20>: failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-rkr4r8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128 -------------------------------------------------------------------------------- RESPONSE 403: 403 Forbidden ERROR CODE: AuthorizationFailed -------------------------------------------------------------------------------- { "error": { "code": "AuthorizationFailed", "message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials." } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-rkr4r8\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0008f2c00>{ msg: "failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-rkr4r8\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0014046f0>{ ErrorCode: "AuthorizationFailed", StatusCode: 403, RawResponse: { Status: "403 Forbidden", StatusCode: 403, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["523"], "X-Ms-Correlation-Request-Id": [ "2935921b-184b-4fc2-9c09-4c26121abf36", ], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Mon, 16 Mar 2026 05:50:31 GMT", ], "Content-Type": [ "application/json; charset=utf-8", ], "Expires": ["-1"], "X-Ms-Failure-Cause": ["gateway"], "X-Ms-Request-Id": [ "2935921b-184b-4fc2-9c09-4c26121abf36", ], "X-Ms-Routing-Request-Id": [ "EASTUS2EUAP:20260316T055032Z:2935921b-184b-4fc2-9c09-4c26121abf36", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error: <fmt.wrapError \| 0xc0008f2c20>: failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-rkr4r8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128 -------------------------------------------------------------------------------- RESPONSE 403: 403 Forbidden ERROR CODE: AuthorizationFailed -------------------------------------------------------------------------------- { "error": { "code": "AuthorizationFailed", "message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials." } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-rkr4r8\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0008f2c00>{ msg: "failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-rkr4r8\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0014046f0>{ ErrorCode: "AuthorizationFailed", StatusCode: 403, RawResponse: { Status: "403 Forbidden", StatusCode: 403, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["523"], "X-Ms-Correlation-Request-Id": [ "2935921b-184b-4fc2-9c09-4c26121abf36", ], "X-Cache": ["CONFIG_NOCACHE"], "Date": [ "Mon, 16 Mar 2026 05:50:31 GMT", ], "Content-Type": [ "application/json; charset=utf-8", ], "Expires": ["-1"], "X-Ms-Failure-Cause": ["gateway"], "X-Ms-Request-Id": [ "2935921b-184b-4fc2-9c09-4c26121abf36", ], "X-Ms-Routing-Request-Id": [ "EASTUS2EUAP:20260316T055032Z:2935921b-184b-4fc2-9c09-4c26121abf36", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 05:06:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error: <fmt.wrapErrors \| 0xc000ddc570>: failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-fsqvnh" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: context deadline exceeded { msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-fsqvnh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0013818e0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh", }, <fmt.wrapErrors \| 0xc000ddc540>{ msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-fsqvnh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0013818e0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error: <fmt.wrapErrors \| 0xc000ddc570>: failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-fsqvnh" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: context deadline exceeded { msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-fsqvnh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0013818e0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh", }, <fmt.wrapErrors \| 0xc000ddc540>{ msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-fsqvnh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0013818e0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh", }, <context.deadlineExceededError>{}, ], }, ], } occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 05:06:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error: <fmt.wrapErrors \| 0xc000e27920>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-m559f4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-m559f4, error: context deadline exceeded { msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-m559f4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-m559f4, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000741990>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-m559f4", }, <context.deadlineExceededError>{}, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error: <fmt.wrapErrors \| 0xc000e27920>: failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-m559f4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-m559f4, error: context deadline exceeded { msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-m559f4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-m559f4, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000741990>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-m559f4", }, <context.deadlineExceededError>{}, ], } occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 05:06:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapError \| 0xc00095c4c0>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-psrj8k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b", "name": "b4b3429d-a139-425f-b110-f68346a4fe8b", "status": "Failed", "startTime": "2026-03-16T05:54:04.320816036Z", "endTime": "2026-03-16T05:54:48.517209203Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-psrj8k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b\",\n \"name\": \"b4b3429d-a139-425f-b110-f68346a4fe8b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:54:04.320816036Z\",\n \"endTime\": \"2026-03-16T05:54:48.517209203Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00095c4a0>{ msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-psrj8k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b\",\n \"name\": \"b4b3429d-a139-425f-b110-f68346a4fe8b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:54:04.320816036Z\",\n \"endTime\": \"2026-03-16T05:54:48.517209203Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000974690>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["503"], "X-Envoy-Upstream-Service-Time": ["10"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Content-Type-Options": ["nosniff"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "39e1d21e-9cf9-413f-8981-bd67e2dcc0da", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Correlation-Request-Id": [ "6c4b46e2-9a40-4232-9e7e-8cba90b18fb2", ], "Pragma": ["no-cache"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/57085293-a6f4-4a48-a611-c5bc14532134", ], "Strict-Transport-Security": [ "m... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error: <fmt.wrapError \| 0xc00095c4c0>: failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-psrj8k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b", "name": "b4b3429d-a139-425f-b110-f68346a4fe8b", "status": "Failed", "startTime": "2026-03-16T05:54:04.320816036Z", "endTime": "2026-03-16T05:54:48.517209203Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-psrj8k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b\",\n \"name\": \"b4b3429d-a139-425f-b110-f68346a4fe8b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:54:04.320816036Z\",\n \"endTime\": \"2026-03-16T05:54:48.517209203Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00095c4a0>{ msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-psrj8k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b\",\n \"name\": \"b4b3429d-a139-425f-b110-f68346a4fe8b\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:54:04.320816036Z\",\n \"endTime\": \"2026-03-16T05:54:48.517209203Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000974690>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Content-Length": ["503"], "X-Envoy-Upstream-Service-Time": ["10"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Content-Type-Options": ["nosniff"], "Content-Type": ["application/json"], "X-Ms-Request-Id": [ "39e1d21e-9cf9-413f-8981-bd67e2dcc0da", ], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Correlation-Request-Id": [ "6c4b46e2-9a40-4232-9e7e-8cba90b18fb2", ], "Pragma": ["no-cache"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/57085293-a6f4-4a48-a611-c5bc14532134", ], "Strict-Transport-Security": [ "m... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 05:06:22 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc001220440>: failed to create HCP cluster cluster-zstream-4-19-slfr7d: failed waiting for cluster="cluster-zstream-4-19-slfr7d" in resourcegroup="rg-zstream-upgrade-4-19-wkgsx5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226", "name": "bbd60cd6-e44d-4a7e-a230-ef5e2ef66226", "status": "Failed", "startTime": "2026-03-16T06:04:10.39702426Z", "endTime": "2026-03-16T06:05:14.283265287Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-slfr7d: failed waiting for cluster=\"cluster-zstream-4-19-slfr7d\" in resourcegroup=\"rg-zstream-upgrade-4-19-wkgsx5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\",\n \"name\": \"bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T06:04:10.39702426Z\",\n \"endTime\": \"2026-03-16T06:05:14.283265287Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001220420>{ msg: "failed waiting for cluster=\"cluster-zstream-4-19-slfr7d\" in resourcegroup=\"rg-zstream-upgrade-4-19-wkgsx5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\",\n \"name\": \"bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T06:04:10.39702426Z\",\n \"endTime\": \"2026-03-16T06:05:14.283265287Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001388720>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Pragma": ["no-cache"], "Content-Length": ["502"], "Content-Type": ["application/json"], "Expires": ["-1"], "X-Ms-Request-Id": [ "3df2300f-9029-4412-ba6b-ef329877e1a7", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/2b9b79d1-95be-4d60-9c5c-30c7bd925b0e", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Envoy-Upstream-Service-Time": ["8"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Correlation-Request-Id": [ "6a3bc74e-3765-48a4-a8c2-26ff92d85256", ], "X-Ms-Routing-Request-Id": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapError \| 0xc001220440>: failed to create HCP cluster cluster-zstream-4-19-slfr7d: failed waiting for cluster="cluster-zstream-4-19-slfr7d" in resourcegroup="rg-zstream-upgrade-4-19-wkgsx5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226", "name": "bbd60cd6-e44d-4a7e-a230-ef5e2ef66226", "status": "Failed", "startTime": "2026-03-16T06:04:10.39702426Z", "endTime": "2026-03-16T06:05:14.283265287Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster cluster-zstream-4-19-slfr7d: failed waiting for cluster=\"cluster-zstream-4-19-slfr7d\" in resourcegroup=\"rg-zstream-upgrade-4-19-wkgsx5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\",\n \"name\": \"bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T06:04:10.39702426Z\",\n \"endTime\": \"2026-03-16T06:05:14.283265287Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001220420>{ msg: "failed waiting for cluster=\"cluster-zstream-4-19-slfr7d\" in resourcegroup=\"rg-zstream-upgrade-4-19-wkgsx5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\",\n \"name\": \"bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T06:04:10.39702426Z\",\n \"endTime\": \"2026-03-16T06:05:14.283265287Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc001388720>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Pragma": ["no-cache"], "Content-Length": ["502"], "Content-Type": ["application/json"], "Expires": ["-1"], "X-Ms-Request-Id": [ "3df2300f-9029-4412-ba6b-ef329877e1a7", ], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/2b9b79d1-95be-4d60-9c5c-30c7bd925b0e", ], "X-Cache": ["CONFIG_NOCACHE"], "X-Envoy-Upstream-Service-Time": ["8"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Correlation-Request-Id": [ "6a3bc74e-3765-48a4-a8c2-26ff92d85256", ], "X-Ms-Routing-Request-Id": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 05:06:22 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error: <fmt.wrapErrors \| 0xc000a4cea0>: failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-b5b8bw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: context deadline exceeded { msg: "failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-b5b8bw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0015a98c0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw", }, <fmt.wrapErrors \| 0xc000a4cae0>{ msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-b5b8bw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0015a98c0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error: <fmt.wrapErrors \| 0xc000a4cea0>: failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-b5b8bw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: context deadline exceeded { msg: "failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-b5b8bw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0015a98c0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw", }, <fmt.wrapErrors \| 0xc000a4cae0>{ msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-b5b8bw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0015a98c0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw", }, <context.deadlineExceededError>{}, ], }, ], } occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 05:06:22 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error: <fmt.wrapError \| 0xc0007a0f20>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-j5p24k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc", "name": "2d2e9797-4414-473e-b4af-5fa3295c39cc", "status": "Failed", "startTime": "2026-03-16T05:58:43.117719355Z", "endTime": "2026-03-16T05:59:25.785868644Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-j5p24k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc\",\n \"name\": \"2d2e9797-4414-473e-b4af-5fa3295c39cc\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:58:43.117719355Z\",\n \"endTime\": \"2026-03-16T05:59:25.785868644Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0007a0ee0>{ msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-j5p24k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc\",\n \"name\": \"2d2e9797-4414-473e-b4af-5fa3295c39cc\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:58:43.117719355Z\",\n \"endTime\": \"2026-03-16T05:59:25.785868644Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000ea4b10>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Expires": ["-1"], "X-Ms-Request-Id": [ "f54f88d5-3cde-4dbc-af86-37ec91b2f129", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Correlation-Request-Id": [ "b6dae1aa-ff73-4fd1-815f-9bb691862adb", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260316T055934Z:b6dae1aa-ff73-4fd1-815f-9bb691862adb", ], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/9a86779f-f19f-4090-82c7-e45be2465d60", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: 2F3009F4FC7845318C8D158FD658CC85 Ref B: CHI2611... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error: <fmt.wrapError \| 0xc0007a0f20>: failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-j5p24k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc", "name": "2d2e9797-4414-473e-b4af-5fa3295c39cc", "status": "Failed", "startTime": "2026-03-16T05:58:43.117719355Z", "endTime": "2026-03-16T05:59:25.785868644Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-j5p24k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc\",\n \"name\": \"2d2e9797-4414-473e-b4af-5fa3295c39cc\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:58:43.117719355Z\",\n \"endTime\": \"2026-03-16T05:59:25.785868644Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc0007a0ee0>{ msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-j5p24k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc\",\n \"name\": \"2d2e9797-4414-473e-b4af-5fa3295c39cc\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:58:43.117719355Z\",\n \"endTime\": \"2026-03-16T05:59:25.785868644Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000ea4b10>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Expires": ["-1"], "X-Ms-Request-Id": [ "f54f88d5-3cde-4dbc-af86-37ec91b2f129", ], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Correlation-Request-Id": [ "b6dae1aa-ff73-4fd1-815f-9bb691862adb", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260316T055934Z:b6dae1aa-ff73-4fd1-815f-9bb691862adb", ], "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/9a86779f-f19f-4090-82c7-e45be2465d60", ], "X-Content-Type-Options": ["nosniff"], "X-Msedge-Ref": [ "Ref A: 2F3009F4FC7845318C8D158FD658CC85 Ref B: CHI2611... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 05:06:22 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error: <fmt.wrapErrors \| 0xc00103ca80>: failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-6m66gz" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: context deadline exceeded { msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-6m66gz\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0008dddb0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz", }, <fmt.wrapErrors \| 0xc00103ca50>{ msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-6m66gz\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0008dddb0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error: <fmt.wrapErrors \| 0xc00103ca80>: failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-6m66gz" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: context deadline exceeded { msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-6m66gz\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0008dddb0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz", }, <fmt.wrapErrors \| 0xc00103ca50>{ msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-6m66gz\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc0008dddb0>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz", }, <context.deadlineExceededError>{}, ], }, ], } occurred ERROR CODE: AuthorizationFailed 05:06:22 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:81]: Unexpected error: <fmt.wrapError \| 0xc001504fc0>: failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster-7hz8zk" in resourcegroup="customer-rg-clsh4k" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609 -------------------------------------------------------------------------------- RESPONSE 403: 403 Forbidden ERROR CODE: AuthorizationFailed -------------------------------------------------------------------------------- { "error": { "code": "AuthorizationFailed", "message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials." } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster-7hz8zk\" in resourcegroup=\"customer-rg-clsh4k\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001504fa0>{ msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster-7hz8zk\" in resourcegroup=\"customer-rg-clsh4k\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0014bb1a0>{ ErrorCode: "AuthorizationFailed", StatusCode: 403, RawResponse: { Status: "403 Forbidden", StatusCode: 403, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Pragma": ["no-cache"], "Expires": ["-1"], "X-Ms-Failure-Cause": ["gateway"], "X-Ms-Correlation-Request-Id": [ "06490acc-bbf6-4428-8144-e61db004ace3", ], "X-Ms-Routing-Request-Id": [ "EASTUS2EUAP:20260316T060848Z:06490acc-bbf6-4428-8144-e61db004ace3", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "Cache-Control": ["no-cache"], "Content-Length": ["569"], "Content-Type": [ "applic... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:81]: Unexpected error: <fmt.wrapError \| 0xc001504fc0>: failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster-7hz8zk" in resourcegroup="customer-rg-clsh4k" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609 -------------------------------------------------------------------------------- RESPONSE 403: 403 Forbidden ERROR CODE: AuthorizationFailed -------------------------------------------------------------------------------- { "error": { "code": "AuthorizationFailed", "message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials." } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster-7hz8zk\" in resourcegroup=\"customer-rg-clsh4k\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc001504fa0>{ msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster-7hz8zk\" in resourcegroup=\"customer-rg-clsh4k\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n \"error\": {\n \"code\": \"AuthorizationFailed\",\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0014bb1a0>{ ErrorCode: "AuthorizationFailed", StatusCode: 403, RawResponse: { Status: "403 Forbidden", StatusCode: 403, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Pragma": ["no-cache"], "Expires": ["-1"], "X-Ms-Failure-Cause": ["gateway"], "X-Ms-Correlation-Request-Id": [ "06490acc-bbf6-4428-8144-e61db004ace3", ], "X-Ms-Routing-Request-Id": [ "EASTUS2EUAP:20260316T060848Z:06490acc-bbf6-4428-8144-e61db004ace3", ], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Content-Type-Options": ["nosniff"], "X-Cache": ["CONFIG_NOCACHE"], "Cache-Control": ["no-cache"], "Content-Length": ["569"], "Content-Type": [ "applic... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code AuthorizationFailed; detail message The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network 05:06:22 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:74]: Unexpected error: <fmt.wrapError \| 0xc00093d060>: failed to create customer-infra: failed waiting for deployment "customer-infra-np-autoscale-cluster-5pp8qh" in resourcegroup="np-autoscaling-l7brcq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Forbidden", "message": "{\r\n \"error\": {\r\n \"code\": \"AuthorizationFailed\",\r\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-np-autoscale-cluster-5pp8qh\" in resourcegroup=\"np-autoscaling-l7brcq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00093d020>{ msg: "failed waiting for deployment \"customer-infra-np-autoscale-cluster-5pp8qh\" in resourcegroup=\"np-autoscaling-l7brcq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0008efc50>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Msedge-Ref": [ "Ref A: ED37EB9E1FA645FA954684A176851865 Ref B: CH1AA2020620045 Ref C: 2026-03-16T06:05:39Z", ], "Date": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:74]: Unexpected error: <fmt.wrapError \| 0xc00093d060>: failed to create customer-infra: failed waiting for deployment "customer-infra-np-autoscale-cluster-5pp8qh" in resourcegroup="np-autoscaling-l7brcq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Forbidden", "message": "{\r\n \"error\": {\r\n \"code\": \"AuthorizationFailed\",\r\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-np-autoscale-cluster-5pp8qh\" in resourcegroup=\"np-autoscaling-l7brcq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00093d020>{ msg: "failed waiting for deployment \"customer-infra-np-autoscale-cluster-5pp8qh\" in resourcegroup=\"np-autoscaling-l7brcq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0008efc50>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Msedge-Ref": [ "Ref A: ED37EB9E1FA645FA954684A176851865 Ref B: CH1AA2020620045 Ref C: 2026-03-16T06:05:39Z", ], "Date": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 05:06:22 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:83]: Unexpected error: <fmt.wrapErrors \| 0xc00144c090>: failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-l82rz5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: context deadline exceeded { msg: "failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-l82rz5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000aed740>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5", }, <fmt.wrapErrors \| 0xc00144c060>{ msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-l82rz5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000aed740>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:83]: Unexpected error: <fmt.wrapErrors \| 0xc00144c090>: failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-l82rz5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: context deadline exceeded { msg: "failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-l82rz5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000aed740>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5", }, <fmt.wrapErrors \| 0xc00144c060>{ msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-l82rz5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000aed740>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5", }, <context.deadlineExceededError>{}, ], }, ], } occurred ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 05:06:22 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error: <fmt.wrapError \| 0xc00088aac0>: failed to create HCP cluster ea-listnpt7gm: failed waiting for cluster="ea-listnpt7gm" in resourcegroup="ea-list-zq4lft" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508", "name": "09a3be35-a51e-474a-8388-36b58cf61508", "status": "Failed", "startTime": "2026-03-16T06:00:58.543305109Z", "endTime": "2026-03-16T06:01:21.462955449Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-listnpt7gm: failed waiting for cluster=\"ea-listnpt7gm\" in resourcegroup=\"ea-list-zq4lft\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508\",\n \"name\": \"09a3be35-a51e-474a-8388-36b58cf61508\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T06:00:58.543305109Z\",\n \"endTime\": \"2026-03-16T06:01:21.462955449Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00088aaa0>{ msg: "failed waiting for cluster=\"ea-listnpt7gm\" in resourcegroup=\"ea-list-zq4lft\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508\",\n \"name\": \"09a3be35-a51e-474a-8388-36b58cf61508\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T06:00:58.543305109Z\",\n \"endTime\": \"2026-03-16T06:01:21.462955449Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00077bbc0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Content-Type": ["application/json"], "X-Envoy-Upstream-Service-Time": ["10"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "Expires": ["-1"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/381463b8-de5e-41cf-a1d6-b149c681d8c1", ], "X-Ms-Correlation-Request-Id": [ "a99d0b49-b6e9-413e-830a-2dcdf4b4a62d", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260316T060129Z:a99d0b49-b6e9-413e-830a-2dcdf4b4a62d", ], "X-Msedge-Ref": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error: <fmt.wrapError \| 0xc00088aac0>: failed to create HCP cluster ea-listnpt7gm: failed waiting for cluster="ea-listnpt7gm" in resourcegroup="ea-list-zq4lft" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508", "name": "09a3be35-a51e-474a-8388-36b58cf61508", "status": "Failed", "startTime": "2026-03-16T06:00:58.543305109Z", "endTime": "2026-03-16T06:01:21.462955449Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster ea-listnpt7gm: failed waiting for cluster=\"ea-listnpt7gm\" in resourcegroup=\"ea-list-zq4lft\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508\",\n \"name\": \"09a3be35-a51e-474a-8388-36b58cf61508\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T06:00:58.543305109Z\",\n \"endTime\": \"2026-03-16T06:01:21.462955449Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00088aaa0>{ msg: "failed waiting for cluster=\"ea-listnpt7gm\" in resourcegroup=\"ea-list-zq4lft\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508\",\n \"name\": \"09a3be35-a51e-474a-8388-36b58cf61508\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T06:00:58.543305109Z\",\n \"endTime\": \"2026-03-16T06:01:21.462955449Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc00077bbc0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Content-Type": ["application/json"], "X-Envoy-Upstream-Service-Time": ["10"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], "Expires": ["-1"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/381463b8-de5e-41cf-a1d6-b149c681d8c1", ], "X-Ms-Correlation-Request-Id": [ "a99d0b49-b6e9-413e-830a-2dcdf4b4a62d", ], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260316T060129Z:a99d0b49-b6e9-413e-830a-2dcdf4b4a62d", ], "X-Msedge-Ref": [ ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred ERROR CODE: DeploymentFailed; detail code AuthorizationFailed; detail message The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network 05:06:22 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error: <fmt.wrapError \| 0xc000760480>: failed to create customer-infra: failed waiting for deployment "customer-infra-basic-cluster-wk5cgh" in resourcegroup="rg-cluster-nsg-subnet-reuse-dvvbjq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Forbidden", "message": "{\r\n \"error\": {\r\n \"code\": \"AuthorizationFailed\",\r\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-cluster-wk5cgh\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-dvvbjq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000760460>{ msg: "failed waiting for deployment \"customer-infra-basic-cluster-wk5cgh\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-dvvbjq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012bd110>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Content-Length": ["839"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error: <fmt.wrapError \| 0xc000760480>: failed to create customer-infra: failed waiting for deployment "customer-infra-basic-cluster-wk5cgh" in resourcegroup="rg-cluster-nsg-subnet-reuse-dvvbjq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Forbidden", "message": "{\r\n \"error\": {\r\n \"code\": \"AuthorizationFailed\",\r\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-cluster-wk5cgh\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-dvvbjq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000760460>{ msg: "failed waiting for deployment \"customer-infra-basic-cluster-wk5cgh\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-dvvbjq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc0012bd110>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Cache-Control": ["no-cache"], "Pragma": ["no-cache"], "Content-Length": ["839"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred Cluster should become ready within 45 minutes 05:06:22 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:157]: Timed out after 2700.001s. Cluster should become ready within 45 minutes Expected <bool>: false to be true fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:157]: Timed out after 2700.001s. Cluster should become ready within 45 minutes Expected <bool>: false to be true ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift 05:06:22 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc00122f260>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-jm9qgg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f", "name": "6998cd22-db46-45f1-aa33-43755a778f3f", "status": "Failed", "startTime": "2026-03-16T05:57:41.628783027Z", "endTime": "2026-03-16T05:58:24.131373366Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-jm9qgg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f\",\n \"name\": \"6998cd22-db46-45f1-aa33-43755a778f3f\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:57:41.628783027Z\",\n \"endTime\": \"2026-03-16T05:58:24.131373366Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00122f240>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-jm9qgg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f\",\n \"name\": \"6998cd22-db46-45f1-aa33-43755a778f3f\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:57:41.628783027Z\",\n \"endTime\": \"2026-03-16T05:58:24.131373366Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000915ad0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Correlation-Request-Id": [ "8e59cc34-80cf-4dd7-a7a4-e8bba1de4a86", ], "Date": [ "Mon, 16 Mar 2026 05:58:32 GMT", ], "Pragma": ["no-cache"], "Expires": ["-1"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/8c536465-a5dc-4067-a393-690861875e85", ], "X-Cache": ["CONFIG_NOCACHE"], "Cache-Control": ["no-cache"], "Content-Length": ["503"], "X-Envoy-Upstream-Service-Time": ["10"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260316T055832Z:8e59cc34-80cf-4dd7-a7a4-e8bba... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error: <fmt.wrapError \| 0xc00122f260>: failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-jm9qgg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: InternalServerError -------------------------------------------------------------------------------- { "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f", "name": "6998cd22-db46-45f1-aa33-43755a778f3f", "status": "Failed", "startTime": "2026-03-16T05:57:41.628783027Z", "endTime": "2026-03-16T05:58:24.131373366Z", "error": { "code": "InternalServerError", "message": "insufficient public IP address quota: required 2, available 0" } } -------------------------------------------------------------------------------- { msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-jm9qgg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f\",\n \"name\": \"6998cd22-db46-45f1-aa33-43755a778f3f\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:57:41.628783027Z\",\n \"endTime\": \"2026-03-16T05:58:24.131373366Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc00122f240>{ msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-jm9qgg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f\",\n \"name\": \"6998cd22-db46-45f1-aa33-43755a778f3f\",\n \"status\": \"Failed\",\n \"startTime\": \"2026-03-16T05:57:41.628783027Z\",\n \"endTime\": \"2026-03-16T05:58:24.131373366Z\",\n \"error\": {\n \"code\": \"InternalServerError\",\n \"message\": \"insufficient public IP address quota: required 2, available 0\"\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000915ad0>{ ErrorCode: "InternalServerError", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"], "X-Ms-Correlation-Request-Id": [ "8e59cc34-80cf-4dd7-a7a4-e8bba1de4a86", ], "Date": [ "Mon, 16 Mar 2026 05:58:32 GMT", ], "Pragma": ["no-cache"], "Expires": ["-1"], "X-Ms-Operation-Identifier": [ "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/8c536465-a5dc-4067-a393-690861875e85", ], "X-Cache": ["CONFIG_NOCACHE"], "Cache-Control": ["no-cache"], "Content-Length": ["503"], "X-Envoy-Upstream-Service-Time": ["10"], "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"], "X-Ms-Routing-Request-Id": [ "CANADACENTRAL:20260316T055832Z:8e59cc34-80cf-4dd7-a7a4-e8bba... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred timeout during CreateHCPClusterAndWait; context deadline exceeded 05:06:22 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapErrors \| 0xc00155d560>: failed to create HCP cluster cluster-zstream-4-20-vrd9tg, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: failed waiting for cluster="cluster-zstream-4-20-vrd9tg" in resourcegroup="rg-zstream-upgrade-4-20-vppdwr" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: context deadline exceeded { msg: "failed to create HCP cluster cluster-zstream-4-20-vrd9tg, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: failed waiting for cluster=\"cluster-zstream-4-20-vrd9tg\" in resourcegroup=\"rg-zstream-upgrade-4-20-vppdwr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000a12410>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr", }, <fmt.wrapErrors \| 0xc00155d530>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-vrd9tg\" in resourcegroup=\"rg-zstream-upgrade-4-20-vppdwr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000a12410>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr", }, <context.deadlineExceededError>{}, ], }, ], } occurred fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error: <fmt.wrapErrors \| 0xc00155d560>: failed to create HCP cluster cluster-zstream-4-20-vrd9tg, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: failed waiting for cluster="cluster-zstream-4-20-vrd9tg" in resourcegroup="rg-zstream-upgrade-4-20-vppdwr" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: context deadline exceeded { msg: "failed to create HCP cluster cluster-zstream-4-20-vrd9tg, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: failed waiting for cluster=\"cluster-zstream-4-20-vrd9tg\" in resourcegroup=\"rg-zstream-upgrade-4-20-vppdwr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000a12410>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr", }, <fmt.wrapErrors \| 0xc00155d530>{ msg: "failed waiting for cluster=\"cluster-zstream-4-20-vrd9tg\" in resourcegroup=\"rg-zstream-upgrade-4-20-vppdwr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: context deadline exceeded", errs: [ <errors.errorString \| 0xc000a12410>{ s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr", }, <context.deadlineExceededError>{}, ], }, ], } occurred ERROR CODE: DeploymentFailed; detail code AuthorizationFailed; detail message The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network 05:06:22 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:84]: Unexpected error: <fmt.wrapError \| 0xc0004993c0>: failed to create customer-infra: failed waiting for deployment "customer-infra-ea-cluster-lqkgx2" in resourcegroup="external-auth-cluster-rhfph2" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Forbidden", "message": "{\r\n \"error\": {\r\n \"code\": \"AuthorizationFailed\",\r\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-ea-cluster-lqkgx2\" in resourcegroup=\"external-auth-cluster-rhfph2\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000499380>{ msg: "failed waiting for deployment \"customer-infra-ea-cluster-lqkgx2\" in resourcegroup=\"external-auth-cluster-rhfph2\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000baa6f0>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:84]: Unexpected error: <fmt.wrapError \| 0xc0004993c0>: failed to create customer-infra: failed waiting for deployment "customer-infra-ea-cluster-lqkgx2" in resourcegroup="external-auth-cluster-rhfph2" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979 -------------------------------------------------------------------------------- RESPONSE 200: 200 OK ERROR CODE: DeploymentFailed -------------------------------------------------------------------------------- { "status": "Failed", "error": { "code": "DeploymentFailed", "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.", "details": [ { "code": "Forbidden", "message": "{\r\n \"error\": {\r\n \"code\": \"AuthorizationFailed\",\r\n \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n }\r\n}" } ] } } -------------------------------------------------------------------------------- { msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-ea-cluster-lqkgx2\" in resourcegroup=\"external-auth-cluster-rhfph2\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <fmt.wrapError \| 0xc000499380>{ msg: "failed waiting for deployment \"customer-infra-ea-cluster-lqkgx2\" in resourcegroup=\"external-auth-cluster-rhfph2\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n \"status\": \"Failed\",\n \"error\": {\n \"code\": \"DeploymentFailed\",\n \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n \"details\": [\n {\n \"code\": \"Forbidden\",\n \"message\": \"{\\r\\n \\\"error\\\": {\\r\\n \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n }\\r\\n}\"\n }\n ]\n }\n}\n--------------------------------------------------------------------------------\n", err: <exported.ResponseError \| 0xc000baa6f0>{ ErrorCode: "DeploymentFailed", StatusCode: 200, RawResponse: { Status: "200 OK", StatusCode: 200, Proto: "HTTP/2.0", ProtoMajor: 2, ProtoMinor: 0, Header: { "Strict-Transport-Security": [ "max-age=31536000; includeSubDomains", ], "X-Cache": ["CONFIG_NOCACHE"], ... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output occurred
02:00:10	periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel post-goodmerged PR No additional run metadata captured	e2e	failed	n/a	1	e2e (1) subscription openshift-nfd/nfd not ready, state="" (OLM still processing, no errors detected) 02:00:10 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel Show raw failure fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:60]: Timed out after 600.007s. NFD operator should be installed successfully Expected success, but got an error: <errors.errorString \| 0xc001039e40>: subscription openshift-nfd/nfd not ready, state="" (OLM still processing, no errors detected) { s: "subscription openshift-nfd/nfd not ready, state=\"\" (OLM still processing, no errors detected)", } fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:60]: Timed out after 600.007s. NFD operator should be installed successfully Expected success, but got an error: <errors.errorString \| 0xc001039e40>: subscription openshift-nfd/nfd not ready, state="" (OLM still processing, no errors detected) { s: "subscription openshift-nfd/nfd not ready, state=\"\" (OLM still processing, no errors detected)", }

Time (UTC)

Job

Failed at

Result

Failed tests

Details

05:06:22

periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel

post-goodmerged PR

No additional run metadata captured

e2e

failed

n/a

e2e (22)

timeout during CreateHCPClusterAndWait; context deadline exceeded

05:06:22 UTC · failed at e2e · test Authorized CIDRs Connectivity should allow API access only from authorized VM IP · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014454d0>: 
    failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-5lns4q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-5lns4q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001459930>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q",
            },
            <*fmt.wrapErrors | 0xc0014454a0>{
                msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-5lns4q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001459930>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_authorized_cidrs_connectivity.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc0014454d0>: 
    failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: failed waiting for cluster="cidr-connectivity-test" in resourcegroup="e2e-cidr-connectivity-5lns4q" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cidr-connectivity-test, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-5lns4q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc001459930>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q",
            },
            <*fmt.wrapErrors | 0xc0014454a0>{
                msg: "failed waiting for cluster=\"cidr-connectivity-test\" in resourcegroup=\"e2e-cidr-connectivity-5lns4q\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc001459930>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cidr-connectivity-test in resource group e2e-cidr-connectivity-5lns4q",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

05:06:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc00101cd20>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-lpp22g" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-lpp22g\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0008a2ae0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g",
            },
            <*fmt.wrapErrors | 0xc00101ccf0>{
                msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-lpp22g\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0008a2ae0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:133]: Unexpected error:
    <*fmt.wrapErrors | 0xc00101cd20>: 
    failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: failed waiting for cluster="pullsecret-hcp-cluster" in resourcegroup="pullsecret-test-lpp22g" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster pullsecret-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-lpp22g\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0008a2ae0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g",
            },
            <*fmt.wrapErrors | 0xc00101ccf0>{
                msg: "failed waiting for cluster=\"pullsecret-hcp-cluster\" in resourcegroup=\"pullsecret-test-lpp22g\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0008a2ae0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster pullsecret-hcp-cluster in resource group pullsecret-test-lpp22g",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

05:06:22 UTC · failed at e2e · test Update HCPOpenShiftCluster Positive creates a cluster and updates tags with a PATCH request · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
    <*fmt.wrapError | 0xc0016292c0>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-6dmknb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487",
      "name": "d3e5222f-7626-4089-81ad-74ce5593b487",
      "status": "Failed",
      "startTime": "2026-03-16T05:53:52.774104349Z",
      "endTime": "2026-03-16T05:54:48.4817526Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-6dmknb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487\",\n  \"name\": \"d3e5222f-7626-4089-81ad-74ce5593b487\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:53:52.774104349Z\",\n  \"endTime\": \"2026-03-16T05:54:48.4817526Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0016292a0>{
            msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-6dmknb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487\",\n  \"name\": \"d3e5222f-7626-4089-81ad-74ce5593b487\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:53:52.774104349Z\",\n  \"endTime\": \"2026-03-16T05:54:48.4817526Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc001384930>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Type": ["application/json"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Content-Type-Options": ["nosniff"],
                        "Date": [
                            "Mon, 16 Mar 2026 05:54:53 GMT",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/4a777810-db65-4717-9cdc-6501be453685",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260316T055453Z:b23331dc-7a87-4aac-85ec-6b1435706e75",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["501"],
                        "X-Envoy-Upstream-Service-Time": ["10"],
 ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:160]: Unexpected error:
    <*fmt.wrapError | 0xc0016292c0>: 
    failed to create HCP cluster patch-tags-cluster: failed waiting for cluster="patch-tags-cluster" in resourcegroup="patch-tags-6dmknb" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487",
      "name": "d3e5222f-7626-4089-81ad-74ce5593b487",
      "status": "Failed",
      "startTime": "2026-03-16T05:53:52.774104349Z",
      "endTime": "2026-03-16T05:54:48.4817526Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster patch-tags-cluster: failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-6dmknb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487\",\n  \"name\": \"d3e5222f-7626-4089-81ad-74ce5593b487\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:53:52.774104349Z\",\n  \"endTime\": \"2026-03-16T05:54:48.4817526Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0016292a0>{
            msg: "failed waiting for cluster=\"patch-tags-cluster\" in resourcegroup=\"patch-tags-6dmknb\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/d3e5222f-7626-4089-81ad-74ce5593b487\",\n  \"name\": \"d3e5222f-7626-4089-81ad-74ce5593b487\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:53:52.774104349Z\",\n  \"endTime\": \"2026-03-16T05:54:48.4817526Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc001384930>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Type": ["application/json"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Content-Type-Options": ["nosniff"],
                        "Date": [
                            "Mon, 16 Mar 2026 05:54:53 GMT",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/4a777810-db65-4717-9cdc-6501be453685",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260316T055453Z:b23331dc-7a87-4aac-85ec-6b1435706e75",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["501"],
                        "X-Envoy-Upstream-Service-Time": ["10"],
 ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

05:06:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error:
    <*fmt.wrapError | 0xc000695540>: 
    failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-vgmwdp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2",
      "name": "8acc7e04-af91-4136-83d9-cfd4d620d5c2",
      "status": "Failed",
      "startTime": "2026-03-16T05:58:36.120361686Z",
      "endTime": "2026-03-16T05:59:25.865971075Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vgmwdp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2\",\n  \"name\": \"8acc7e04-af91-4136-83d9-cfd4d620d5c2\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:58:36.120361686Z\",\n  \"endTime\": \"2026-03-16T05:59:25.865971075Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000695500>{
            msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vgmwdp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2\",\n  \"name\": \"8acc7e04-af91-4136-83d9-cfd4d620d5c2\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:58:36.120361686Z\",\n  \"endTime\": \"2026-03-16T05:59:25.865971075Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000adeed0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "20c01661-0afe-48d2-8f91-e191ffbc139d",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260316T055926Z:6229d6a4-a060-4bcd-8adb-767c2b1c2524",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["11"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/f93d9d56-0328-4fdd-b9c9-60f270da1dcd",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "Strict-Transport-Security": [
                            "max-age=3153...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/complete_cluster_create.go:96]: Unexpected error:
    <*fmt.wrapError | 0xc000695540>: 
    failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster="basic-hcp-cluster" in resourcegroup="basic-cluster-vgmwdp" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2",
      "name": "8acc7e04-af91-4136-83d9-cfd4d620d5c2",
      "status": "Failed",
      "startTime": "2026-03-16T05:58:36.120361686Z",
      "endTime": "2026-03-16T05:59:25.865971075Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster basic-hcp-cluster: failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vgmwdp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2\",\n  \"name\": \"8acc7e04-af91-4136-83d9-cfd4d620d5c2\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:58:36.120361686Z\",\n  \"endTime\": \"2026-03-16T05:59:25.865971075Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000695500>{
            msg: "failed waiting for cluster=\"basic-hcp-cluster\" in resourcegroup=\"basic-cluster-vgmwdp\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/8acc7e04-af91-4136-83d9-cfd4d620d5c2\",\n  \"name\": \"8acc7e04-af91-4136-83d9-cfd4d620d5c2\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:58:36.120361686Z\",\n  \"endTime\": \"2026-03-16T05:59:25.865971075Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000adeed0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "20c01661-0afe-48d2-8f91-e191ffbc139d",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260316T055926Z:6229d6a4-a060-4bcd-8adb-767c2b1c2524",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["11"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/f93d9d56-0328-4fdd-b9c9-60f270da1dcd",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "Strict-Transport-Security": [
                            "max-age=3153...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

05:06:22 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.21 · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapErrors | 0xc00106e6f0>: 
    failed to create HCP cluster cluster-zstream-4-21-jsjz55, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: failed waiting for cluster="cluster-zstream-4-21-jsjz55" in resourcegroup="rg-zstream-upgrade-4-21-b2bt9k" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cluster-zstream-4-21-jsjz55, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: failed waiting for cluster=\"cluster-zstream-4-21-jsjz55\" in resourcegroup=\"rg-zstream-upgrade-4-21-b2bt9k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a91e10>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k",
            },
            <*fmt.wrapErrors | 0xc00106e6c0>{
                msg: "failed waiting for cluster=\"cluster-zstream-4-21-jsjz55\" in resourcegroup=\"rg-zstream-upgrade-4-21-b2bt9k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000a91e10>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapErrors | 0xc00106e6f0>: 
    failed to create HCP cluster cluster-zstream-4-21-jsjz55, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: failed waiting for cluster="cluster-zstream-4-21-jsjz55" in resourcegroup="rg-zstream-upgrade-4-21-b2bt9k" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cluster-zstream-4-21-jsjz55, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: failed waiting for cluster=\"cluster-zstream-4-21-jsjz55\" in resourcegroup=\"rg-zstream-upgrade-4-21-b2bt9k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a91e10>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k",
            },
            <*fmt.wrapErrors | 0xc00106e6c0>{
                msg: "failed waiting for cluster=\"cluster-zstream-4-21-jsjz55\" in resourcegroup=\"rg-zstream-upgrade-4-21-b2bt9k\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000a91e10>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-21-jsjz55 in resource group rg-zstream-upgrade-4-21-b2bt9k",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

ERROR CODE: AuthorizationFailed; detail message The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '<uuid>' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.RedHatOpenShift

05:06:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster and custom node pool osDisk size · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapError | 0xc0008f2c20>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-rkr4r8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 403: 403 Forbidden
    ERROR CODE: AuthorizationFailed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "AuthorizationFailed",
        "message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-rkr4r8\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0008f2c00>{
            msg: "failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-rkr4r8\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0014046f0>{
                ErrorCode: "AuthorizationFailed",
                StatusCode: 403,
                RawResponse: {
                    Status: "403 Forbidden",
                    StatusCode: 403,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["523"],
                        "X-Ms-Correlation-Request-Id": [
                            "2935921b-184b-4fc2-9c09-4c26121abf36",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Date": [
                            "Mon, 16 Mar 2026 05:50:31 GMT",
                        ],
                        "Content-Type": [
                            "application/json; charset=utf-8",
                        ],
                        "Expires": ["-1"],
                        "X-Ms-Failure-Cause": ["gateway"],
                        "X-Ms-Request-Id": [
                            "2935921b-184b-4fc2-9c09-4c26121abf36",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "EASTUS2EUAP:20260316T055032Z:2935921b-184b-4fc2-9c09-4c26121abf36",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
    ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_create_nodepool_osdisk.go:80]: Unexpected error:
    <*fmt.wrapError | 0xc0008f2c20>: 
    failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation "hcp-cluster-np-128" in resourcegroup="clusternp128-rkr4r8": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128
    --------------------------------------------------------------------------------
    RESPONSE 403: 403 Forbidden
    ERROR CODE: AuthorizationFailed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "AuthorizationFailed",
        "message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster hcp-cluster-np-128: failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-rkr4r8\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0008f2c00>{
            msg: "failed starting cluster creation \"hcp-cluster-np-128\" in resourcegroup=\"clusternp128-rkr4r8\": PUT https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.RedHatOpenShift/hcpOpenShiftClusters/write' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourceGroups/clusternp128-rkr4r8/providers/Microsoft.RedHatOpenShift/hcpOpenShiftClusters/hcp-cluster-np-128' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0014046f0>{
                ErrorCode: "AuthorizationFailed",
                StatusCode: 403,
                RawResponse: {
                    Status: "403 Forbidden",
                    StatusCode: 403,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["523"],
                        "X-Ms-Correlation-Request-Id": [
                            "2935921b-184b-4fc2-9c09-4c26121abf36",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Date": [
                            "Mon, 16 Mar 2026 05:50:31 GMT",
                        ],
                        "Content-Type": [
                            "application/json; charset=utf-8",
                        ],
                        "Expires": ["-1"],
                        "X-Ms-Failure-Cause": ["gateway"],
                        "X-Ms-Request-Id": [
                            "2935921b-184b-4fc2-9c09-4c26121abf36",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "EASTUS2EUAP:20260316T055032Z:2935921b-184b-4fc2-9c09-4c26121abf36",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
    ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

05:06:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster with custom autoscaling · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ddc570>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-fsqvnh" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-fsqvnh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0013818e0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh",
            },
            <*fmt.wrapErrors | 0xc000ddc540>{
                msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-fsqvnh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0013818e0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_autoscaling.go:97]: Unexpected error:
    <*fmt.wrapErrors | 0xc000ddc570>: 
    failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: failed waiting for cluster="autoscaling-hcp-cluster" in resourcegroup="autoscaling-cluster-fsqvnh" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster autoscaling-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-fsqvnh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0013818e0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh",
            },
            <*fmt.wrapErrors | 0xc000ddc540>{
                msg: "failed waiting for cluster=\"autoscaling-hcp-cluster\" in resourcegroup=\"autoscaling-cluster-fsqvnh\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0013818e0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster autoscaling-hcp-cluster in resource group autoscaling-cluster-fsqvnh",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

05:06:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster with back-level version 4.19 · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e27920>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-m559f4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-m559f4, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-m559f4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-m559f4, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000741990>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-m559f4",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_version_backlevel.go:148]: Unexpected error:
    <*fmt.wrapErrors | 0xc000e27920>: 
    failed waiting for cluster="cluster-ver-4-19" in resourcegroup="rg-cluster-back-version-m559f4" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-m559f4, error: context deadline exceeded
    {
        msg: "failed waiting for cluster=\"cluster-ver-4-19\" in resourcegroup=\"rg-cluster-back-version-m559f4\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-m559f4, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000741990>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterAndWait for cluster cluster-ver-4-19 in resource group rg-cluster-back-version-m559f4",
            },
            <context.deadlineExceededError>{},
        ],
    }
occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

05:06:22 UTC · failed at e2e · test Customer should be able to create an HCP cluster with Image Registry not present · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapError | 0xc00095c4c0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-psrj8k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b",
      "name": "b4b3429d-a139-425f-b110-f68346a4fe8b",
      "status": "Failed",
      "startTime": "2026-03-16T05:54:04.320816036Z",
      "endTime": "2026-03-16T05:54:48.517209203Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-psrj8k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b\",\n  \"name\": \"b4b3429d-a139-425f-b110-f68346a4fe8b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:54:04.320816036Z\",\n  \"endTime\": \"2026-03-16T05:54:48.517209203Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00095c4a0>{
            msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-psrj8k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b\",\n  \"name\": \"b4b3429d-a139-425f-b110-f68346a4fe8b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:54:04.320816036Z\",\n  \"endTime\": \"2026-03-16T05:54:48.517209203Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000974690>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["10"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Content-Type-Options": ["nosniff"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "39e1d21e-9cf9-413f-8981-bd67e2dcc0da",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Correlation-Request-Id": [
                            "6c4b46e2-9a40-4232-9e7e-8cba90b18fb2",
                        ],
                        "Pragma": ["no-cache"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/57085293-a6f4-4a48-a611-c5bc14532134",
                        ],
                        "Strict-Transport-Security": [
                            "m...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/image_registry_cluster_create.go:89]: Unexpected error:
    <*fmt.wrapError | 0xc00095c4c0>: 
    failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster="disabled-image-registry-hcp-cluster" in resourcegroup="disabled-image-registry-psrj8k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b",
      "name": "b4b3429d-a139-425f-b110-f68346a4fe8b",
      "status": "Failed",
      "startTime": "2026-03-16T05:54:04.320816036Z",
      "endTime": "2026-03-16T05:54:48.517209203Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster disabled-image-registry-hcp-cluster: failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-psrj8k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b\",\n  \"name\": \"b4b3429d-a139-425f-b110-f68346a4fe8b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:54:04.320816036Z\",\n  \"endTime\": \"2026-03-16T05:54:48.517209203Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00095c4a0>{
            msg: "failed waiting for cluster=\"disabled-image-registry-hcp-cluster\" in resourcegroup=\"disabled-image-registry-psrj8k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/b4b3429d-a139-425f-b110-f68346a4fe8b\",\n  \"name\": \"b4b3429d-a139-425f-b110-f68346a4fe8b\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:54:04.320816036Z\",\n  \"endTime\": \"2026-03-16T05:54:48.517209203Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000974690>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["10"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Content-Type-Options": ["nosniff"],
                        "Content-Type": ["application/json"],
                        "X-Ms-Request-Id": [
                            "39e1d21e-9cf9-413f-8981-bd67e2dcc0da",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Correlation-Request-Id": [
                            "6c4b46e2-9a40-4232-9e7e-8cba90b18fb2",
                        ],
                        "Pragma": ["no-cache"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/57085293-a6f4-4a48-a611-c5bc14532134",
                        ],
                        "Strict-Transport-Security": [
                            "m...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

05:06:22 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.19 · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc001220440>: 
    failed to create HCP cluster cluster-zstream-4-19-slfr7d: failed waiting for cluster="cluster-zstream-4-19-slfr7d" in resourcegroup="rg-zstream-upgrade-4-19-wkgsx5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226",
      "name": "bbd60cd6-e44d-4a7e-a230-ef5e2ef66226",
      "status": "Failed",
      "startTime": "2026-03-16T06:04:10.39702426Z",
      "endTime": "2026-03-16T06:05:14.283265287Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-slfr7d: failed waiting for cluster=\"cluster-zstream-4-19-slfr7d\" in resourcegroup=\"rg-zstream-upgrade-4-19-wkgsx5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\",\n  \"name\": \"bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T06:04:10.39702426Z\",\n  \"endTime\": \"2026-03-16T06:05:14.283265287Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc001220420>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-19-slfr7d\" in resourcegroup=\"rg-zstream-upgrade-4-19-wkgsx5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\",\n  \"name\": \"bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T06:04:10.39702426Z\",\n  \"endTime\": \"2026-03-16T06:05:14.283265287Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc001388720>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Pragma": ["no-cache"],
                        "Content-Length": ["502"],
                        "Content-Type": ["application/json"],
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "3df2300f-9029-4412-ba6b-ef329877e1a7",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/2b9b79d1-95be-4d60-9c5c-30c7bd925b0e",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Envoy-Upstream-Service-Time": ["8"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Correlation-Request-Id": [
                            "6a3bc74e-3765-48a4-a8c2-26ff92d85256",
                        ],
                        "X-Ms-Routing-Request-Id": [
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapError | 0xc001220440>: 
    failed to create HCP cluster cluster-zstream-4-19-slfr7d: failed waiting for cluster="cluster-zstream-4-19-slfr7d" in resourcegroup="rg-zstream-upgrade-4-19-wkgsx5" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226",
      "name": "bbd60cd6-e44d-4a7e-a230-ef5e2ef66226",
      "status": "Failed",
      "startTime": "2026-03-16T06:04:10.39702426Z",
      "endTime": "2026-03-16T06:05:14.283265287Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster cluster-zstream-4-19-slfr7d: failed waiting for cluster=\"cluster-zstream-4-19-slfr7d\" in resourcegroup=\"rg-zstream-upgrade-4-19-wkgsx5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\",\n  \"name\": \"bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T06:04:10.39702426Z\",\n  \"endTime\": \"2026-03-16T06:05:14.283265287Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc001220420>{
            msg: "failed waiting for cluster=\"cluster-zstream-4-19-slfr7d\" in resourcegroup=\"rg-zstream-upgrade-4-19-wkgsx5\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\",\n  \"name\": \"bbd60cd6-e44d-4a7e-a230-ef5e2ef66226\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T06:04:10.39702426Z\",\n  \"endTime\": \"2026-03-16T06:05:14.283265287Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc001388720>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Pragma": ["no-cache"],
                        "Content-Length": ["502"],
                        "Content-Type": ["application/json"],
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "3df2300f-9029-4412-ba6b-ef329877e1a7",
                        ],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/2b9b79d1-95be-4d60-9c5c-30c7bd925b0e",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "X-Envoy-Upstream-Service-Time": ["8"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Correlation-Request-Id": [
                            "6a3bc74e-3765-48a4-a8c2-26ff92d85256",
                        ],
                        "X-Ms-Routing-Request-Id": [
                        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

05:06:22 UTC · failed at e2e · test Update HCPOpenShiftCluster Negative creates a cluster and fails to update its name with a PATCH request · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a4cea0>: 
    failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-b5b8bw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-b5b8bw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0015a98c0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw",
            },
            <*fmt.wrapErrors | 0xc000a4cae0>{
                msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-b5b8bw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0015a98c0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_update.go:84]: Unexpected error:
    <*fmt.wrapErrors | 0xc000a4cea0>: 
    failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: failed waiting for cluster="patch-name-cluster" in resourcegroup="patch-name-b5b8bw" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster patch-name-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-b5b8bw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0015a98c0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw",
            },
            <*fmt.wrapErrors | 0xc000a4cae0>{
                msg: "failed waiting for cluster=\"patch-name-cluster\" in resourcegroup=\"patch-name-b5b8bw\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0015a98c0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster patch-name-cluster in resource group patch-name-b5b8bw",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

05:06:22 UTC · failed at e2e · test Customer should not be able to perform various invalid operations on cluster resources · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error:
    <*fmt.wrapError | 0xc0007a0f20>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-j5p24k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc",
      "name": "2d2e9797-4414-473e-b4af-5fa3295c39cc",
      "status": "Failed",
      "startTime": "2026-03-16T05:58:43.117719355Z",
      "endTime": "2026-03-16T05:59:25.785868644Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-j5p24k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc\",\n  \"name\": \"2d2e9797-4414-473e-b4af-5fa3295c39cc\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:58:43.117719355Z\",\n  \"endTime\": \"2026-03-16T05:59:25.785868644Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0007a0ee0>{
            msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-j5p24k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc\",\n  \"name\": \"2d2e9797-4414-473e-b4af-5fa3295c39cc\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:58:43.117719355Z\",\n  \"endTime\": \"2026-03-16T05:59:25.785868644Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000ea4b10>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "f54f88d5-3cde-4dbc-af86-37ec91b2f129",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Correlation-Request-Id": [
                            "b6dae1aa-ff73-4fd1-815f-9bb691862adb",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260316T055934Z:b6dae1aa-ff73-4fd1-815f-9bb691862adb",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/9a86779f-f19f-4090-82c7-e45be2465d60",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Msedge-Ref": [
                            "Ref A: 2F3009F4FC7845318C8D158FD658CC85 Ref B: CHI2611...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/simple_negative_cases.go:89]: Unexpected error:
    <*fmt.wrapError | 0xc0007a0f20>: 
    failed to create HCP cluster negative-tests-cluster: failed waiting for cluster="negative-tests-cluster" in resourcegroup="rg-negative-tests-j5p24k" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc",
      "name": "2d2e9797-4414-473e-b4af-5fa3295c39cc",
      "status": "Failed",
      "startTime": "2026-03-16T05:58:43.117719355Z",
      "endTime": "2026-03-16T05:59:25.785868644Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster negative-tests-cluster: failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-j5p24k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc\",\n  \"name\": \"2d2e9797-4414-473e-b4af-5fa3295c39cc\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:58:43.117719355Z\",\n  \"endTime\": \"2026-03-16T05:59:25.785868644Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc0007a0ee0>{
            msg: "failed waiting for cluster=\"negative-tests-cluster\" in resourcegroup=\"rg-negative-tests-j5p24k\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/2d2e9797-4414-473e-b4af-5fa3295c39cc\",\n  \"name\": \"2d2e9797-4414-473e-b4af-5fa3295c39cc\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:58:43.117719355Z\",\n  \"endTime\": \"2026-03-16T05:59:25.785868644Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000ea4b10>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Expires": ["-1"],
                        "X-Ms-Request-Id": [
                            "f54f88d5-3cde-4dbc-af86-37ec91b2f129",
                        ],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Correlation-Request-Id": [
                            "b6dae1aa-ff73-4fd1-815f-9bb691862adb",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260316T055934Z:b6dae1aa-ff73-4fd1-815f-9bb691862adb",
                        ],
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/9a86779f-f19f-4090-82c7-e45be2465d60",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Msedge-Ref": [
                            "Ref A: 2F3009F4FC7845318C8D158FD658CC85 Ref B: CHI2611...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

05:06:22 UTC · failed at e2e · test Customer should be able to create node pools with ARM64-based VMs · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc00103ca80>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-6m66gz" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-6m66gz\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0008dddb0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz",
            },
            <*fmt.wrapErrors | 0xc00103ca50>{
                msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-6m66gz\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0008dddb0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/arm64_nodepool.go:90]: Unexpected error:
    <*fmt.wrapErrors | 0xc00103ca80>: 
    failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: failed waiting for cluster="arm64-vm-hcp-cluster" in resourcegroup="arm64-vm-cluster-6m66gz" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster arm64-vm-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-6m66gz\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc0008dddb0>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz",
            },
            <*fmt.wrapErrors | 0xc00103ca50>{
                msg: "failed waiting for cluster=\"arm64-vm-hcp-cluster\" in resourcegroup=\"arm64-vm-cluster-6m66gz\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc0008dddb0>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster arm64-vm-hcp-cluster in resource group arm64-vm-cluster-6m66gz",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

ERROR CODE: AuthorizationFailed

05:06:22 UTC · failed at e2e · test Customer should be able to create several HCP clusters in their customer resource group, but not in the same managed resource group · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:81]: Unexpected error:
    <*fmt.wrapError | 0xc001504fc0>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster-7hz8zk" in resourcegroup="customer-rg-clsh4k" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609
    --------------------------------------------------------------------------------
    RESPONSE 403: 403 Forbidden
    ERROR CODE: AuthorizationFailed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "AuthorizationFailed",
        "message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster-7hz8zk\" in resourcegroup=\"customer-rg-clsh4k\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc001504fa0>{
            msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster-7hz8zk\" in resourcegroup=\"customer-rg-clsh4k\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0014bb1a0>{
                ErrorCode: "AuthorizationFailed",
                StatusCode: 403,
                RawResponse: {
                    Status: "403 Forbidden",
                    StatusCode: 403,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Failure-Cause": ["gateway"],
                        "X-Ms-Correlation-Request-Id": [
                            "06490acc-bbf6-4428-8144-e61db004ace3",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "EASTUS2EUAP:20260316T060848Z:06490acc-bbf6-4428-8144-e61db004ace3",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["569"],
                        "Content-Type": [
                            "applic...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/clusters_sharing_resgroup.go:81]: Unexpected error:
    <*fmt.wrapError | 0xc001504fc0>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-hcp-cluster-7hz8zk" in resourcegroup="customer-rg-clsh4k" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609
    --------------------------------------------------------------------------------
    RESPONSE 403: 403 Forbidden
    ERROR CODE: AuthorizationFailed
    --------------------------------------------------------------------------------
    {
      "error": {
        "code": "AuthorizationFailed",
        "message": "The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials."
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-hcp-cluster-7hz8zk\" in resourcegroup=\"customer-rg-clsh4k\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc001504fa0>{
            msg: "failed waiting for deployment \"customer-infra-basic-hcp-cluster-7hz8zk\" in resourcegroup=\"customer-rg-clsh4k\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609\n--------------------------------------------------------------------------------\nRESPONSE 403: 403 Forbidden\nERROR CODE: AuthorizationFailed\n--------------------------------------------------------------------------------\n{\n  \"error\": {\n    \"code\": \"AuthorizationFailed\",\n    \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Resources/deployments/operationStatuses/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/customer-rg-clsh4k/providers/Microsoft.Resources/deployments/customer-infra-basic-hcp-cluster-7hz8zk/operationStatuses/08584279655981786609' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0014bb1a0>{
                ErrorCode: "AuthorizationFailed",
                StatusCode: 403,
                RawResponse: {
                    Status: "403 Forbidden",
                    StatusCode: 403,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Failure-Cause": ["gateway"],
                        "X-Ms-Correlation-Request-Id": [
                            "06490acc-bbf6-4428-8144-e61db004ace3",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "EASTUS2EUAP:20260316T060848Z:06490acc-bbf6-4428-8144-e61db004ace3",
                        ],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Content-Type-Options": ["nosniff"],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["569"],
                        "Content-Type": [
                            "applic...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: DeploymentFailed; detail code AuthorizationFailed; detail message The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network

05:06:22 UTC · failed at e2e · test Customer should be able to create a cluster with default autoscaling and a nodepool with autoscaling enabled · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:74]: Unexpected error:
    <*fmt.wrapError | 0xc00093d060>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-np-autoscale-cluster-5pp8qh" in resourcegroup="np-autoscaling-l7brcq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Forbidden",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"AuthorizationFailed\",\r\n    \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-np-autoscale-cluster-5pp8qh\" in resourcegroup=\"np-autoscaling-l7brcq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Forbidden\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n    \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00093d020>{
            msg: "failed waiting for deployment \"customer-infra-np-autoscale-cluster-5pp8qh\" in resourcegroup=\"np-autoscaling-l7brcq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Forbidden\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n    \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0008efc50>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Msedge-Ref": [
                            "Ref A: ED37EB9E1FA645FA954684A176851865 Ref B: CH1AA2020620045 Ref C: 2026-03-16T06:05:39Z",
                        ],
                        "Date": [
        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:74]: Unexpected error:
    <*fmt.wrapError | 0xc00093d060>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-np-autoscale-cluster-5pp8qh" in resourcegroup="np-autoscaling-l7brcq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Forbidden",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"AuthorizationFailed\",\r\n    \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-np-autoscale-cluster-5pp8qh\" in resourcegroup=\"np-autoscaling-l7brcq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Forbidden\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n    \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00093d020>{
            msg: "failed waiting for deployment \"customer-infra-np-autoscale-cluster-5pp8qh\" in resourcegroup=\"np-autoscaling-l7brcq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Resources/deployments/customer-infra-np-autoscale-cluster-5pp8qh/operationStatuses/08584279658982549597\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Forbidden\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n    \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/np-autoscaling-l7brcq/providers/Microsoft.Network/virtualNetworks/customer-vnet' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0008efc50>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Msedge-Ref": [
                            "Ref A: ED37EB9E1FA645FA954684A176851865 Ref B: CH1AA2020620045 Ref C: 2026-03-16T06:05:39Z",
                        ],
                        "Date": [
        ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

05:06:22 UTC · failed at e2e · test Customer should create an HCP cluster and validate TLS certificates · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:83]: Unexpected error:
    <*fmt.wrapErrors | 0xc00144c090>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-l82rz5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-l82rz5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000aed740>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5",
            },
            <*fmt.wrapErrors | 0xc00144c060>{
                msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-l82rz5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000aed740>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_tls_endpoints.go:83]: Unexpected error:
    <*fmt.wrapErrors | 0xc00144c090>: 
    failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: failed waiting for cluster="tls-endpoint-hcp-cluster" in resourcegroup="tls-endpoint-cluster-l82rz5" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster tls-endpoint-hcp-cluster, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-l82rz5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000aed740>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5",
            },
            <*fmt.wrapErrors | 0xc00144c060>{
                msg: "failed waiting for cluster=\"tls-endpoint-hcp-cluster\" in resourcegroup=\"tls-endpoint-cluster-l82rz5\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000aed740>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster tls-endpoint-hcp-cluster in resource group tls-endpoint-cluster-l82rz5",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

05:06:22 UTC · failed at e2e · test Customer should be able to lifecycle and confirm external auth on a cluster · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapError | 0xc00088aac0>: 
    failed to create HCP cluster ea-listnpt7gm: failed waiting for cluster="ea-listnpt7gm" in resourcegroup="ea-list-zq4lft" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508",
      "name": "09a3be35-a51e-474a-8388-36b58cf61508",
      "status": "Failed",
      "startTime": "2026-03-16T06:00:58.543305109Z",
      "endTime": "2026-03-16T06:01:21.462955449Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ea-listnpt7gm: failed waiting for cluster=\"ea-listnpt7gm\" in resourcegroup=\"ea-list-zq4lft\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508\",\n  \"name\": \"09a3be35-a51e-474a-8388-36b58cf61508\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T06:00:58.543305109Z\",\n  \"endTime\": \"2026-03-16T06:01:21.462955449Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00088aaa0>{
            msg: "failed waiting for cluster=\"ea-listnpt7gm\" in resourcegroup=\"ea-list-zq4lft\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508\",\n  \"name\": \"09a3be35-a51e-474a-8388-36b58cf61508\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T06:00:58.543305109Z\",\n  \"endTime\": \"2026-03-16T06:01:21.462955449Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00077bbc0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Envoy-Upstream-Service-Time": ["10"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Expires": ["-1"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/381463b8-de5e-41cf-a1d6-b149c681d8c1",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "a99d0b49-b6e9-413e-830a-2dcdf4b4a62d",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260316T060129Z:a99d0b49-b6e9-413e-830a-2dcdf4b4a62d",
                        ],
                        "X-Msedge-Ref": [
  ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_list_and_verify.go:84]: Unexpected error:
    <*fmt.wrapError | 0xc00088aac0>: 
    failed to create HCP cluster ea-listnpt7gm: failed waiting for cluster="ea-listnpt7gm" in resourcegroup="ea-list-zq4lft" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508",
      "name": "09a3be35-a51e-474a-8388-36b58cf61508",
      "status": "Failed",
      "startTime": "2026-03-16T06:00:58.543305109Z",
      "endTime": "2026-03-16T06:01:21.462955449Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster ea-listnpt7gm: failed waiting for cluster=\"ea-listnpt7gm\" in resourcegroup=\"ea-list-zq4lft\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508\",\n  \"name\": \"09a3be35-a51e-474a-8388-36b58cf61508\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T06:00:58.543305109Z\",\n  \"endTime\": \"2026-03-16T06:01:21.462955449Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00088aaa0>{
            msg: "failed waiting for cluster=\"ea-listnpt7gm\" in resourcegroup=\"ea-list-zq4lft\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/09a3be35-a51e-474a-8388-36b58cf61508\",\n  \"name\": \"09a3be35-a51e-474a-8388-36b58cf61508\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T06:00:58.543305109Z\",\n  \"endTime\": \"2026-03-16T06:01:21.462955449Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc00077bbc0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "Content-Type": ["application/json"],
                        "X-Envoy-Upstream-Service-Time": ["10"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Expires": ["-1"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/381463b8-de5e-41cf-a1d6-b149c681d8c1",
                        ],
                        "X-Ms-Correlation-Request-Id": [
                            "a99d0b49-b6e9-413e-830a-2dcdf4b4a62d",
                        ],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260316T060129Z:a99d0b49-b6e9-413e-830a-2dcdf4b4a62d",
                        ],
                        "X-Msedge-Ref": [
  ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

ERROR CODE: DeploymentFailed; detail code AuthorizationFailed; detail message The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network

05:06:22 UTC · failed at e2e · test Customer should not be able to reuse subnets and NSGs between clusters · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error:
    <*fmt.wrapError | 0xc000760480>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-cluster-wk5cgh" in resourcegroup="rg-cluster-nsg-subnet-reuse-dvvbjq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Forbidden",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"AuthorizationFailed\",\r\n    \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-cluster-wk5cgh\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-dvvbjq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Forbidden\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n    \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000760460>{
            msg: "failed waiting for deployment \"customer-infra-basic-cluster-wk5cgh\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-dvvbjq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Forbidden\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n    \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0012bd110>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "Content-Length": ["839"],
                 ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_nsg_subnet_reuse.go:96]: Unexpected error:
    <*fmt.wrapError | 0xc000760480>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-basic-cluster-wk5cgh" in resourcegroup="rg-cluster-nsg-subnet-reuse-dvvbjq" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Forbidden",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"AuthorizationFailed\",\r\n    \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-basic-cluster-wk5cgh\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-dvvbjq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Forbidden\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n    \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000760460>{
            msg: "failed waiting for deployment \"customer-infra-basic-cluster-wk5cgh\" in resourcegroup=\"rg-cluster-nsg-subnet-reuse-dvvbjq\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Resources/deployments/customer-infra-basic-cluster-wk5cgh/operationStatuses/08584279668585153729\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Forbidden\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n    \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/virtualNetworks/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/rg-cluster-nsg-subnet-reuse-dvvbjq/providers/Microsoft.Network/virtualNetworks/customer-vnet-name1' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc0012bd110>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Cache-Control": ["no-cache"],
                        "Pragma": ["no-cache"],
                        "Content-Length": ["839"],
                 ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

Cluster should become ready within 45 minutes

05:06:22 UTC · failed at e2e · test Customer should be able to test admin credentials before cluster ready, then full admin credential lifecycle · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:157]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true
fail [github.com/Azure/ARO-HCP/test/e2e/admin_credential_lifecycle.go:157]: Timed out after 2700.001s.
Cluster should become ready within 45 minutes
Expected
    <bool>: false
to be true

ERROR CODE: InternalServerError; detail message insufficient public IP address quota: required <count>, available <count>; provider Microsoft.RedHatOpenShift

05:06:22 UTC · failed at e2e · test Customer should respect cluster-wide node limits with nodepool autoscaling · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapError | 0xc00122f260>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-jm9qgg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f",
      "name": "6998cd22-db46-45f1-aa33-43755a778f3f",
      "status": "Failed",
      "startTime": "2026-03-16T05:57:41.628783027Z",
      "endTime": "2026-03-16T05:58:24.131373366Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-jm9qgg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f\",\n  \"name\": \"6998cd22-db46-45f1-aa33-43755a778f3f\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:57:41.628783027Z\",\n  \"endTime\": \"2026-03-16T05:58:24.131373366Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00122f240>{
            msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-jm9qgg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f\",\n  \"name\": \"6998cd22-db46-45f1-aa33-43755a778f3f\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:57:41.628783027Z\",\n  \"endTime\": \"2026-03-16T05:58:24.131373366Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000915ad0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Correlation-Request-Id": [
                            "8e59cc34-80cf-4dd7-a7a4-e8bba1de4a86",
                        ],
                        "Date": [
                            "Mon, 16 Mar 2026 05:58:32 GMT",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/8c536465-a5dc-4067-a393-690861875e85",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["10"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260316T055832Z:8e59cc34-80cf-4dd7-a7a4-e8bba...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/nodepool_autoscaling.go:166]: Unexpected error:
    <*fmt.wrapError | 0xc00122f260>: 
    failed to create HCP cluster node-limit-cluster: failed waiting for cluster="node-limit-cluster" in resourcegroup="node-limit-test-jm9qgg" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: InternalServerError
    --------------------------------------------------------------------------------
    {
      "id": "/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f",
      "name": "6998cd22-db46-45f1-aa33-43755a778f3f",
      "status": "Failed",
      "startTime": "2026-03-16T05:57:41.628783027Z",
      "endTime": "2026-03-16T05:58:24.131373366Z",
      "error": {
        "code": "InternalServerError",
        "message": "insufficient public IP address quota: required 2, available 0"
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create HCP cluster node-limit-cluster: failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-jm9qgg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f\",\n  \"name\": \"6998cd22-db46-45f1-aa33-43755a778f3f\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:57:41.628783027Z\",\n  \"endTime\": \"2026-03-16T05:58:24.131373366Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc00122f240>{
            msg: "failed waiting for cluster=\"node-limit-cluster\" in resourcegroup=\"node-limit-test-jm9qgg\" to finish creating: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: InternalServerError\n--------------------------------------------------------------------------------\n{\n  \"id\": \"/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/providers/Microsoft.RedHatOpenShift/locations/eastus2euap/hcpOperationStatuses/6998cd22-db46-45f1-aa33-43755a778f3f\",\n  \"name\": \"6998cd22-db46-45f1-aa33-43755a778f3f\",\n  \"status\": \"Failed\",\n  \"startTime\": \"2026-03-16T05:57:41.628783027Z\",\n  \"endTime\": \"2026-03-16T05:58:24.131373366Z\",\n  \"error\": {\n    \"code\": \"InternalServerError\",\n    \"message\": \"insufficient public IP address quota: required 2, available 0\"\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000915ad0>{
                ErrorCode: "InternalServerError",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "X-Ms-Ratelimit-Remaining-Subscription-Global-Reads": ["3749"],
                        "X-Ms-Correlation-Request-Id": [
                            "8e59cc34-80cf-4dd7-a7a4-e8bba1de4a86",
                        ],
                        "Date": [
                            "Mon, 16 Mar 2026 05:58:32 GMT",
                        ],
                        "Pragma": ["no-cache"],
                        "Expires": ["-1"],
                        "X-Ms-Operation-Identifier": [
                            "appId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,tenantId=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX,objectId=0e96dbb7-17a2-4393-b2d3-d84524e39c32/canadacentral/8c536465-a5dc-4067-a393-690861875e85",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
                        "Cache-Control": ["no-cache"],
                        "Content-Length": ["503"],
                        "X-Envoy-Upstream-Service-Time": ["10"],
                        "X-Ms-Ratelimit-Remaining-Subscription-Reads": ["249"],
                        "X-Ms-Routing-Request-Id": [
                            "CANADACENTRAL:20260316T055832Z:8e59cc34-80cf-4dd7-a7a4-e8bba...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

timeout during CreateHCPClusterAndWait; context deadline exceeded

05:06:22 UTC · failed at e2e · test Control plane automated z-stream upgrade with candidate channel should perform an automated control plane z-stream upgrade for 4.20 · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapErrors | 0xc00155d560>: 
    failed to create HCP cluster cluster-zstream-4-20-vrd9tg, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: failed waiting for cluster="cluster-zstream-4-20-vrd9tg" in resourcegroup="rg-zstream-upgrade-4-20-vppdwr" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cluster-zstream-4-20-vrd9tg, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: failed waiting for cluster=\"cluster-zstream-4-20-vrd9tg\" in resourcegroup=\"rg-zstream-upgrade-4-20-vppdwr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a12410>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr",
            },
            <*fmt.wrapErrors | 0xc00155d530>{
                msg: "failed waiting for cluster=\"cluster-zstream-4-20-vrd9tg\" in resourcegroup=\"rg-zstream-upgrade-4-20-vppdwr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000a12410>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred
fail [github.com/Azure/ARO-HCP/test/e2e/control_plane_automated_z_stream_upgrade.go:95]: Unexpected error:
    <*fmt.wrapErrors | 0xc00155d560>: 
    failed to create HCP cluster cluster-zstream-4-20-vrd9tg, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: failed waiting for cluster="cluster-zstream-4-20-vrd9tg" in resourcegroup="rg-zstream-upgrade-4-20-vppdwr" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: context deadline exceeded
    {
        msg: "failed to create HCP cluster cluster-zstream-4-20-vrd9tg, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: failed waiting for cluster=\"cluster-zstream-4-20-vrd9tg\" in resourcegroup=\"rg-zstream-upgrade-4-20-vppdwr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: context deadline exceeded",
        errs: [
            <*errors.errorString | 0xc000a12410>{
                s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr",
            },
            <*fmt.wrapErrors | 0xc00155d530>{
                msg: "failed waiting for cluster=\"cluster-zstream-4-20-vrd9tg\" in resourcegroup=\"rg-zstream-upgrade-4-20-vppdwr\" to finish creating, caused by: timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr, error: context deadline exceeded",
                errs: [
                    <*errors.errorString | 0xc000a12410>{
                        s: "timeout '45.000000' minutes exceeded during CreateHCPClusterFromParam for cluster cluster-zstream-4-20-vrd9tg in resource group rg-zstream-upgrade-4-20-vppdwr",
                    },
                    <context.deadlineExceededError>{},
                ],
            },
        ],
    }
occurred

ERROR CODE: DeploymentFailed; detail code AuthorizationFailed; detail message The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '<uuid>' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/<resource>' over scope '/subscriptions/<subscription>/resourcegroups/<resource-group>/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.; provider Microsoft.Network

05:06:22 UTC · failed at e2e · test Customer should be able to create a cluster with an external auth config and get the external auth config · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:84]: Unexpected error:
    <*fmt.wrapError | 0xc0004993c0>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-ea-cluster-lqkgx2" in resourcegroup="external-auth-cluster-rhfph2" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Forbidden",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"AuthorizationFailed\",\r\n    \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-ea-cluster-lqkgx2\" in resourcegroup=\"external-auth-cluster-rhfph2\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Forbidden\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n    \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000499380>{
            msg: "failed waiting for deployment \"customer-infra-ea-cluster-lqkgx2\" in resourcegroup=\"external-auth-cluster-rhfph2\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Forbidden\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n    \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000baa6f0>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
     ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred
fail [github.com/Azure/ARO-HCP/test/e2e/external_auth_create.go:84]: Unexpected error:
    <*fmt.wrapError | 0xc0004993c0>: 
    failed to create customer-infra: failed waiting for deployment "customer-infra-ea-cluster-lqkgx2" in resourcegroup="external-auth-cluster-rhfph2" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979
    --------------------------------------------------------------------------------
    RESPONSE 200: 200 OK
    ERROR CODE: DeploymentFailed
    --------------------------------------------------------------------------------
    {
      "status": "Failed",
      "error": {
        "code": "DeploymentFailed",
        "message": "At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.",
        "details": [
          {
            "code": "Forbidden",
            "message": "{\r\n  \"error\": {\r\n    \"code\": \"AuthorizationFailed\",\r\n    \"message\": \"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\"\r\n  }\r\n}"
          }
        ]
      }
    }
    --------------------------------------------------------------------------------
    
    {
        msg: "failed to create customer-infra: failed waiting for deployment \"customer-infra-ea-cluster-lqkgx2\" in resourcegroup=\"external-auth-cluster-rhfph2\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Forbidden\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n    \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
        err: <*fmt.wrapError | 0xc000499380>{
            msg: "failed waiting for deployment \"customer-infra-ea-cluster-lqkgx2\" in resourcegroup=\"external-auth-cluster-rhfph2\" to finish: GET https://management.azure.com/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Resources/deployments/customer-infra-ea-cluster-lqkgx2/operationStatuses/08584279665118667979\n--------------------------------------------------------------------------------\nRESPONSE 200: 200 OK\nERROR CODE: DeploymentFailed\n--------------------------------------------------------------------------------\n{\n  \"status\": \"Failed\",\n  \"error\": {\n    \"code\": \"DeploymentFailed\",\n    \"message\": \"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/arm-deployment-operations for usage details.\",\n    \"details\": [\n      {\n        \"code\": \"Forbidden\",\n        \"message\": \"{\\r\\n  \\\"error\\\": {\\r\\n    \\\"code\\\": \\\"AuthorizationFailed\\\",\\r\\n    \\\"message\\\": \\\"The client 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' with object id '0e96dbb7-17a2-4393-b2d3-d84524e39c32' does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/read' over scope '/subscriptions/XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX/resourcegroups/external-auth-cluster-rhfph2/providers/Microsoft.Network/networkSecurityGroups/customer-nsg' or the scope is invalid. If access was recently granted, please refresh your credentials.\\\"\\r\\n  }\\r\\n}\"\n      }\n    ]\n  }\n}\n--------------------------------------------------------------------------------\n",
            err: <*exported.ResponseError | 0xc000baa6f0>{
                ErrorCode: "DeploymentFailed",
                StatusCode: 200,
                RawResponse: {
                    Status: "200 OK",
                    StatusCode: 200,
                    Proto: "HTTP/2.0",
                    ProtoMajor: 2,
                    ProtoMinor: 0,
                    Header: {
                        "Strict-Transport-Security": [
                            "max-age=31536000; includeSubDomains",
                        ],
                        "X-Cache": ["CONFIG_NOCACHE"],
     ...

Gomega truncated this representation as it exceeds 'format.MaxLength'.
Consider having the object provide a custom 'GomegaStringer' representation
or adjust the parameters in Gomega's 'format' package.

Learn more here: https://onsi.github.io/gomega/#adjusting-output

occurred

02:00:10

periodic-ci-Azure-ARO-HCP-main-periodic-prod-e2e-parallel

post-goodmerged PR

No additional run metadata captured

e2e

failed

n/a

e2e (1)

subscription openshift-nfd/nfd not ready, state="" (OLM still processing, no errors detected)

02:00:10 UTC · failed at e2e · test Customer should be able to create an HCP cluster and manage pull secrets · suite prod/parallel

Show raw failure

fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:60]: Timed out after 600.007s.
NFD operator should be installed successfully
Expected success, but got an error:
    <*errors.errorString | 0xc001039e40>: 
    subscription openshift-nfd/nfd not ready, state="" (OLM still processing, no errors detected)
    {
        s: "subscription openshift-nfd/nfd not ready, state=\"\" (OLM still processing, no errors detected)",
    }
fail [github.com/Azure/ARO-HCP/test/e2e/cluster_pullsecret.go:60]: Timed out after 600.007s.
NFD operator should be installed successfully
Expected success, but got an error:
    <*errors.errorString | 0xc001039e40>: 
    subscription openshift-nfd/nfd not ready, state="" (OLM still processing, no errors detected)
    {
        s: "subscription openshift-nfd/nfd not ready, state=\"\" (OLM still processing, no errors detected)",
    }